LinuxQuestions.org
Help answer threads with 0 replies.
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices


Reply
  Search this Thread
Old 07-24-2008, 02:03 AM   #16
Mr. C.
Senior Member
 
Registered: Jun 2008
Posts: 2,529

Rep: Reputation: 63

Last I looked, scp was not UDP-based, so that breaks your basic UDP requirement.

And NFS??? That just plain makes no sense. How are you going to handle security, hung mounts, excessive retransmits, user and group ids, and generally lackluster performance? I don't think you know what you're getting into here.

If you want performance, blast the bits yourself as others have mentioned, and handle file integrity as necessary. Don't count on bytes transmitted vs. bytes received as an integrity check.
 
Old 07-25-2008, 07:59 AM   #17
David1357
Senior Member
 
Registered: Aug 2007
Location: South Carolina, U.S.A.
Distribution: Ubuntu, Fedora Core, Red Hat, SUSE, Gentoo, DSL, coLinux, uClinux
Posts: 1,302
Blog Entries: 1

Rep: Reputation: 107Reputation: 107
Quote:
Originally Posted by montylee View Post
lol, i had made some changes in the code. I tried transferring a file contains 8000+ lines and it was transferred correctly. Don't know how it is working properly now.
Do you use any kind of version tracking system? If so, you can diff your current version against your previous version. If you are not using a version tracking system, you have just found out why you need to start.

Quote:
Originally Posted by montylee View Post
Here's the code being used for sending the file:
Code:
#define PACKET_SIZE	512
/* Open the file to be sent to the server */
if ((fd = open(deviceInfo->recFile, O_RDONLY)) < 0) {
	sprintf (message, "Error opening file '%s': %s", deviceInfo->recFile, strerror (errno));
	send_msg_to_server (message);
	return NULL;
}
/* Send the file contents to the automation server */
while ((numbytes = read (fd, buf, PACKET_SIZE - 1)) > 0) {
	buf[numbytes] = '\0';
	if ((numbytes = send_msg_to_server (buf)) == -1) {
		close (fd);
		return NULL;
	}
}
close (fd);
I would have used a do/while loop here which would allow you to check the return value of numbytes in a separate statement. On Linux, the return value for an error is -1, but if numbytes is unsigned, it will never be "< 0". Splitting out the check of the return value into separate statements can help you find bugs like that.

Quote:
Originally Posted by montylee View Post
Here's the code for receiving the file:
Code:
#define PACKET_SIZE	512
ufds[0].fd = g_sockfd;
ufds[0].events = POLLIN;
addr_len = sizeof (recv_addr);
/* Read the file contents from the socket and write it to a text file */
while ((retval = poll (ufds, 1, TIMEOUT)) > 0) {
	if ((numbytes = recvfrom (g_sockfd, buf, PACKET_SIZE, 0,
		(struct sockaddr *) &recv_addr, &addr_len)) > 0) {
		buf[numbytes] = '\0';
		if ((numbytes = fwrite(&buf, strlen (buf), 1, fpRec)) <= 0) {
			perror("fwrite");
			return -1;
		}
		fflush(fpRec);
	} else {
		perror("recvfrom");
		return -1;
	}
}
Well, this is exactly what I was warning you about. You have no sanity check on your file chunks. Usually, people create header structures like this
Code:
typedef struct _CHUNK_HEADER
{
    int    iType;        // Allows for sending different chunk types
    int    iSize;        // Allows for validating the size
    int    iChecksum;    // Allows for detection of corrupt chunks
    int    iSequence;    // Allows for detection of missing chunks
} CHUNK_HEADER, *PCHUNK_HEADER __attribute__((packed));
Without some sort of header like this on each chunk, you have no idea whether what you received is what you sent. I have had people tell me that the checksum is unnecessary, but I have seen cases where the IP checksum passed because it does not handle bursts of errors very well. The probability of both checksums failing is very low.

Quote:
Originally Posted by montylee View Post
So, i guess that the sendto() was sending the bytes correctly, but some bytes were getting lost, so recvfrom() wasn't getting the entire data.
If you were using sequence numbers, you would know if you were losing chunks. You really need to beef up your error checking and your "protocol" if you want to figure out why things work intermittently.
 
Old 07-25-2008, 08:52 AM   #18
montylee
Member
 
Registered: May 2003
Location: India
Distribution: Ubuntu 7.04, Fedora Core 9
Posts: 168

Original Poster
Rep: Reputation: 30
Quote:
Originally Posted by Mr. C. View Post
Last I looked, scp was not UDP-based, so that breaks your basic UDP requirement.

And NFS??? That just plain makes no sense. How are you going to handle security, hung mounts, excessive retransmits, user and group ids, and generally lackluster performance? I don't think you know what you're getting into here.

If you want performance, blast the bits yourself as others have mentioned, and handle file integrity as necessary. Don't count on bytes transmitted vs. bytes received as an integrity check.
Thanks for the suggestion! I'll look into what solution i can use.


Quote:
Originally Posted by David1357 View Post
Do you use any kind of version tracking system? If so, you can diff your current version against your previous version. If you are not using a version tracking system, you have just found out why you need to start.
I use Serena Version Manager. Actually i have made numerous changes to the code but no change to the file sending code.



Quote:
Originally Posted by David1357 View Post
I would have used a do/while loop here which would allow you to check the return value of numbytes in a separate statement. On Linux, the return value for an error is -1, but if numbytes is unsigned, it will never be "< 0". Splitting out the check of the return value into separate statements can help you find bugs like that.
Thanks for the suggestion. I'll look into this.


Quote:
Originally Posted by David1357 View Post
Well, this is exactly what I was warning you about. You have no sanity check on your file chunks. Usually, people create header structures like this
Code:
typedef struct _CHUNK_HEADER
{
    int    iType;        // Allows for sending different chunk types
    int    iSize;        // Allows for validating the size
    int    iChecksum;    // Allows for detection of corrupt chunks
    int    iSequence;    // Allows for detection of missing chunks
} CHUNK_HEADER, *PCHUNK_HEADER __attribute__((packed));
Without some sort of header like this on each chunk, you have no idea whether what you received is what you sent. I have had people tell me that the checksum is unnecessary, but I have seen cases where the IP checksum passed because it does not handle bursts of errors very well. The probability of both checksums failing is very low.



If you were using sequence numbers, you would know if you were losing chunks. You really need to beef up your error checking and your "protocol" if you want to figure out why things work intermittently.
Can u please point me to a good tutorial where i can find info about headers and such stuff.

I referred Beej's networking guide and i only know that i can send a string buffer through socket using sendto(). I have no idea about associating headers with the message and decoding it at the receiving end. A link to a tutorial would be of great help.

Last edited by montylee; 07-25-2008 at 08:53 AM.
 
Old 07-25-2008, 04:39 PM   #19
David1357
Senior Member
 
Registered: Aug 2007
Location: South Carolina, U.S.A.
Distribution: Ubuntu, Fedora Core, Red Hat, SUSE, Gentoo, DSL, coLinux, uClinux
Posts: 1,302
Blog Entries: 1

Rep: Reputation: 107Reputation: 107
Quote:
Originally Posted by montylee View Post
Can u please point me to a good tutorial where i can find info about headers and such stuff.
You make your own headers:
Code:
// Use static so that you don't waste the stack
static unsigned char ucBuffer[sizeof(CHUNK_HEADER) + 512];
PCHUNK_HEADER *pHeader = (PCHUNK_HEADER) ucBuffer;

pHeader->iType = FILE_CHUNK;
pHeader->iSize = sizeof(ucBuffer);
for (i = 0; i < iFileLength; i++)
{
    fread(&ucBuffer[sizeof(CHUNK_HEADER)], 512, 1, fd);
    CalculateChecksum(ucBuffer, &pHeader->iSize);
    pHeader->iSequence = i;
    write(sd, ucBuffer, sizeof(ucBuffer));
}
Obviously, this is just skeleton code and I have left out error checking and a lot of the details of handling files and sockets. But this should give you the basic idea of how to add your own headers to the data you send. If it does not, you might want to invest in a good book.
 
Old 07-26-2008, 02:58 AM   #20
montylee
Member
 
Registered: May 2003
Location: India
Distribution: Ubuntu 7.04, Fedora Core 9
Posts: 168

Original Poster
Rep: Reputation: 30
thanks for the sample code David!
Can u please suggest some online links or books where i can find more information regarding this.
 
Old 07-27-2008, 07:50 PM   #21
chrism01
LQ Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Rocky 9.2
Posts: 18,348

Rep: Reputation: 2749Reputation: 2749Reputation: 2749Reputation: 2749Reputation: 2749Reputation: 2749Reputation: 2749Reputation: 2749Reputation: 2749Reputation: 2749Reputation: 2749
If you don't want to do your own checksum calcn, invoke md5sum (man md5sum) and calc that at the sender, send the checksum value as part of the msg eg in the 'header' referred to, then re-calc on the receiver and compare to the sent value.
 
Old 07-28-2008, 10:45 AM   #22
montylee
Member
 
Registered: May 2003
Location: India
Distribution: Ubuntu 7.04, Fedora Core 9
Posts: 168

Original Poster
Rep: Reputation: 30
actually i am already using md5sum but currently i am using it on the entire file. I'll use it per message now.
 
Old 07-28-2008, 12:10 PM   #23
Mr. C.
Senior Member
 
Registered: Jun 2008
Posts: 2,529

Rep: Reputation: 63
There are plenty of open source checksum algorithms that will allow you to perform checksums on chunks. This will allow you to checksum each chunk before dumping it on the wire, and upon receiving it. This may allow you to provide faster failure detection and recovery.
 
Old 08-11-2008, 07:20 AM   #24
David1357
Senior Member
 
Registered: Aug 2007
Location: South Carolina, U.S.A.
Distribution: Ubuntu, Fedora Core, Red Hat, SUSE, Gentoo, DSL, coLinux, uClinux
Posts: 1,302
Blog Entries: 1

Rep: Reputation: 107Reputation: 107
Quote:
Originally Posted by montylee View Post
thanks for the sample code David!
Can u please suggest some online links or books where i can find more information regarding this.
I have never seen code like that in a book. You will find code similar to that in the Linux kernel source. If you do find a book with sample code like that in it, please post the title here for everyone's benefit.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Transfer files from linux to sony ericsson cellphones a.dehqan Linux - Software 4 07-17-2008 05:02 AM
Can I transfer files from win to linux?(Noob) Normal Linux - Networking 14 09-10-2007 12:39 AM
How to transfer files from Linux to Windows SentralOrigin Linux - General 7 10-31-2005 04:29 PM
Software to transfer files from windows to linux b123coder Linux - Software 12 01-30-2005 02:48 PM
how to transfer files from NTFS to EXt3 linux BartekPL Linux - Newbie 4 02-05-2004 05:14 PM

LinuxQuestions.org > Forums > Non-*NIX Forums > Programming

All times are GMT -5. The time now is 09:13 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration