Best way to transfer files in Linux thru C programming?
ProgrammingThis forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
Last I looked, scp was not UDP-based, so that breaks your basic UDP requirement.
And NFS??? That just plain makes no sense. How are you going to handle security, hung mounts, excessive retransmits, user and group ids, and generally lackluster performance? I don't think you know what you're getting into here.
If you want performance, blast the bits yourself as others have mentioned, and handle file integrity as necessary. Don't count on bytes transmitted vs. bytes received as an integrity check.
lol, i had made some changes in the code. I tried transferring a file contains 8000+ lines and it was transferred correctly. Don't know how it is working properly now.
Do you use any kind of version tracking system? If so, you can diff your current version against your previous version. If you are not using a version tracking system, you have just found out why you need to start.
Quote:
Originally Posted by montylee
Here's the code being used for sending the file:
Code:
#define PACKET_SIZE 512
/* Open the file to be sent to the server */
if ((fd = open(deviceInfo->recFile, O_RDONLY)) < 0) {
sprintf (message, "Error opening file '%s': %s", deviceInfo->recFile, strerror (errno));
send_msg_to_server (message);
return NULL;
}
/* Send the file contents to the automation server */
while ((numbytes = read (fd, buf, PACKET_SIZE - 1)) > 0) {
buf[numbytes] = '\0';
if ((numbytes = send_msg_to_server (buf)) == -1) {
close (fd);
return NULL;
}
}
close (fd);
I would have used a do/while loop here which would allow you to check the return value of numbytes in a separate statement. On Linux, the return value for an error is -1, but if numbytes is unsigned, it will never be "< 0". Splitting out the check of the return value into separate statements can help you find bugs like that.
Quote:
Originally Posted by montylee
Here's the code for receiving the file:
Code:
#define PACKET_SIZE 512
ufds[0].fd = g_sockfd;
ufds[0].events = POLLIN;
addr_len = sizeof (recv_addr);
/* Read the file contents from the socket and write it to a text file */
while ((retval = poll (ufds, 1, TIMEOUT)) > 0) {
if ((numbytes = recvfrom (g_sockfd, buf, PACKET_SIZE, 0,
(struct sockaddr *) &recv_addr, &addr_len)) > 0) {
buf[numbytes] = '\0';
if ((numbytes = fwrite(&buf, strlen (buf), 1, fpRec)) <= 0) {
perror("fwrite");
return -1;
}
fflush(fpRec);
} else {
perror("recvfrom");
return -1;
}
}
Well, this is exactly what I was warning you about. You have no sanity check on your file chunks. Usually, people create header structures like this
Code:
typedef struct _CHUNK_HEADER
{
int iType; // Allows for sending different chunk types
int iSize; // Allows for validating the size
int iChecksum; // Allows for detection of corrupt chunks
int iSequence; // Allows for detection of missing chunks
} CHUNK_HEADER, *PCHUNK_HEADER __attribute__((packed));
Without some sort of header like this on each chunk, you have no idea whether what you received is what you sent. I have had people tell me that the checksum is unnecessary, but I have seen cases where the IP checksum passed because it does not handle bursts of errors very well. The probability of both checksums failing is very low.
Quote:
Originally Posted by montylee
So, i guess that the sendto() was sending the bytes correctly, but some bytes were getting lost, so recvfrom() wasn't getting the entire data.
If you were using sequence numbers, you would know if you were losing chunks. You really need to beef up your error checking and your "protocol" if you want to figure out why things work intermittently.
Last I looked, scp was not UDP-based, so that breaks your basic UDP requirement.
And NFS??? That just plain makes no sense. How are you going to handle security, hung mounts, excessive retransmits, user and group ids, and generally lackluster performance? I don't think you know what you're getting into here.
If you want performance, blast the bits yourself as others have mentioned, and handle file integrity as necessary. Don't count on bytes transmitted vs. bytes received as an integrity check.
Thanks for the suggestion! I'll look into what solution i can use.
Quote:
Originally Posted by David1357
Do you use any kind of version tracking system? If so, you can diff your current version against your previous version. If you are not using a version tracking system, you have just found out why you need to start.
I use Serena Version Manager. Actually i have made numerous changes to the code but no change to the file sending code.
Quote:
Originally Posted by David1357
I would have used a do/while loop here which would allow you to check the return value of numbytes in a separate statement. On Linux, the return value for an error is -1, but if numbytes is unsigned, it will never be "< 0". Splitting out the check of the return value into separate statements can help you find bugs like that.
Thanks for the suggestion. I'll look into this.
Quote:
Originally Posted by David1357
Well, this is exactly what I was warning you about. You have no sanity check on your file chunks. Usually, people create header structures like this
Code:
typedef struct _CHUNK_HEADER
{
int iType; // Allows for sending different chunk types
int iSize; // Allows for validating the size
int iChecksum; // Allows for detection of corrupt chunks
int iSequence; // Allows for detection of missing chunks
} CHUNK_HEADER, *PCHUNK_HEADER __attribute__((packed));
Without some sort of header like this on each chunk, you have no idea whether what you received is what you sent. I have had people tell me that the checksum is unnecessary, but I have seen cases where the IP checksum passed because it does not handle bursts of errors very well. The probability of both checksums failing is very low.
If you were using sequence numbers, you would know if you were losing chunks. You really need to beef up your error checking and your "protocol" if you want to figure out why things work intermittently.
Can u please point me to a good tutorial where i can find info about headers and such stuff.
I referred Beej's networking guide and i only know that i can send a string buffer through socket using sendto(). I have no idea about associating headers with the message and decoding it at the receiving end. A link to a tutorial would be of great help.
Can u please point me to a good tutorial where i can find info about headers and such stuff.
You make your own headers:
Code:
// Use static so that you don't waste the stack
static unsigned char ucBuffer[sizeof(CHUNK_HEADER) + 512];
PCHUNK_HEADER *pHeader = (PCHUNK_HEADER) ucBuffer;
pHeader->iType = FILE_CHUNK;
pHeader->iSize = sizeof(ucBuffer);
for (i = 0; i < iFileLength; i++)
{
fread(&ucBuffer[sizeof(CHUNK_HEADER)], 512, 1, fd);
CalculateChecksum(ucBuffer, &pHeader->iSize);
pHeader->iSequence = i;
write(sd, ucBuffer, sizeof(ucBuffer));
}
Obviously, this is just skeleton code and I have left out error checking and a lot of the details of handling files and sockets. But this should give you the basic idea of how to add your own headers to the data you send. If it does not, you might want to invest in a good book.
If you don't want to do your own checksum calcn, invoke md5sum (man md5sum) and calc that at the sender, send the checksum value as part of the msg eg in the 'header' referred to, then re-calc on the receiver and compare to the sent value.
There are plenty of open source checksum algorithms that will allow you to perform checksums on chunks. This will allow you to checksum each chunk before dumping it on the wire, and upon receiving it. This may allow you to provide faster failure detection and recovery.
thanks for the sample code David!
Can u please suggest some online links or books where i can find more information regarding this.
I have never seen code like that in a book. You will find code similar to that in the Linux kernel source. If you do find a book with sample code like that in it, please post the title here for everyone's benefit.
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.