LinuxQuestions.org

LinuxQuestions.org (/questions/)
-   Programming (https://www.linuxquestions.org/questions/programming-9/)
-   -   Best way to transfer files in Linux thru C programming? (https://www.linuxquestions.org/questions/programming-9/best-way-to-transfer-files-in-linux-thru-c-programming-657577/)

montylee 07-22-2008 11:06 PM

Best way to transfer files in Linux thru C programming?
 
Here's my requirement:

I have a server on which some files are present. There are N number of clients. The server sends a command to the clients with a name of a file. The clients need to retrieve the file from the server.

Currently i am just using sockets to send the file contents from the server to client, but this method has a problem. When i transfer a large file, the client doesn't receive the entire file contents i.e. some file contents are lost. The method works for small files though. All my code is in ANSI C.

Now, what is the best possible method to get the files from the server? The server might send different file names to different clients, so each client needs to retrieve the respective file from the server.

I am thinking of using SCP or NFS for file transfer. I need to use SCP or NFS from within my C code, so if the server requires a password for SCP, i need to automate this in my code. For e.g. the user can specify the user name and password in a config file at the client side. i'll just read the config file, get the user name, password from the file and use SCP to get the files from the server.

There are no major security requirements but i think the clients should specify a password before it can get the file from the server.

Basically i need to do this thru my C program. how can i do that? I think i can use SCP or TFTP.

I was thinking of using cURL but i am not sure if i should use it. The clients in my case are embedded machines with limited memory.

Please help me.

Mr. C. 07-23-2008 12:34 AM

You started out developing a sockets-based networking application, but didn't get it to work correctly. So you're scrapping that approach, and are now thinking you should invoke an existing transfer application from within your own?

I don't get it - what's the point now of wrapping an existing file transfer application into another layer?

Could be... a homework assignment you couldn't complete?

Quote:

Originally Posted by montylee
or like u mentioned TFTP.

Like who mentioned? Your instructor?

montylee 07-23-2008 01:23 AM

lol i am not a student anymore :) I am working on a client project in Linux.
That TFTP was mentioned by someone on another forum, so i just copy pasted my thread from that forum. I can give u the link to that forum if you want. I'll edit my post...

Actually my application is still using sockets for other communication stuff. I want to use something else just for file transfer as using sockets is giving me a problem. I am not scrapping sockets or anything. I wanted to know what method i can use for file transfer in a C program.

David1357 07-23-2008 01:48 AM

Quote:

Originally Posted by montylee (Post 3223312)
Currently i am just using sockets to send the file contents from the server to client, but this method has a problem. When i transfer a large file, the client doesn't receive the entire file contents i.e. some file contents are lost. The method works for small files though. All my code is in ANSI C.

I have written code to transfer files in C many times. Are you using UDP or TCP? Are you trying to send the file all at once or are you splitting it up into pieces? The problem is solvable, but depending on your code, it may take a while to debug.

Quote:

Originally Posted by montylee (Post 3223312)
I am thinking of using SCP or NFS for file transfer. I need to use SCP or NFS from within my C code, so if the server requires a password for SCP, i need to automate this in my code.

The only way I have been able to successfully use SCP in a script without shared keys is using expect. For Ubuntu, it can be easily installed using apt or aptitude.

Here is the script I used (Note: The location of the brackets seems to be important)
Code:

#!/usr/bin/expect
spawn scp auser@amachine:~/testfile.txt .
expect "password:" {
        send "apassword\r"
        expect "100%"
}

It took some digging around on the web and some local tinkering to get the magic right.

If you are really space limited and security is not crucial, it may be easier to use FTP. It can be driven directly from a bash script.

Mr. C. 07-23-2008 01:54 AM

Or you can open a pty and fork/exec the transfer app as if it were on a terminal. Still, the method is just plain silly; there are so many other dependencies. To each his own.

theNbomr 07-23-2008 09:31 AM

How about taking an existing file transfer protocol, and embed the code into you application? This way you can use an existing known-good tool on each end of the transfer while developing and debugging the peer file transfer code. There are many options, all of which should have readily available code and documentation that describes the protocol. Many protocols were developed to run on simple machinery, and are not complex to implement.
My sense is that your existing code can probably be debugged and made to work without a huge effort, especially if you are using a known-good TCP stack, and are transferring data using TCP. If the relevant code segment is not huge, perhaps posting it here will let a keen eye spot the problem.
--- rod.

chrism01 07-23-2008 10:02 AM

Makes sense to debug your socket code.
All the higher level solns boil down to a socket (TCP or UDP) connection anyway...
http://en.wikipedia.org/wiki/Internet_layer

montylee 07-23-2008 10:36 AM

thanks for the replies :)
My current code is using UDP. I can't change it to TCP as it's client requirement.

The file i want to transfer has the following format:

id=1,type=1,code=2,value=1,tv_sec=2,tv_usec=222

My file contains multiple lines of the above type. When the number of lines is small, the file is transferred properly but when the number of lines increase, some lines are corrupted or not transferred. Even on a single machine i got this problem.

@David, i am splitting the files into chunks of 512 bytes and transferring them thru UDP socket. Should i try it with smaller size chunks? Should i post the code here? Please suggest...

I read a bit about expect and it is an excellent way of using SCP, but i have one doubt about the same:

I'll create an expect script and call it from my C code. But can i get some return value from the script so that i can get an idea whether the file transfer succeeded or failed? That's my only doubt.

I also thought about using NFS. I can simply mount the server directory containing my data on the clients and access the file directly. This seems to be a good solution.

What do you guys suggest?

Even if NFS or SCP works for me i would like to resolve the file transfer problem i am facing thru sockets.

jiml8 07-23-2008 11:36 AM

UDP is not guaranteed delivery, but on a single machine you certainly shouldn't be having trouble with it. I would guess you have a bad pointer someplace.

To deploy a reliable protocol using UDP over the internet it seems to me you'll have to break your transfer up into packets and tag each packet with a number and a checksum. Also maybe some header with a total number of packets. Then, if any packets arrive corrupted or do not arrive at all, the recipient can request a resend.

Or you could zip the whole thing up and send it via ftp.

I have a commercial app, that I sell, which does pretty much what you want. I use UDP because it is stateless and connectionless and that is how I want it. I have found it to be very advantageous to zip up large text files before sending them, and unzip them at the other end. Faster that way, and less likely to lose a packet that has to be resent.

David1357 07-23-2008 01:19 PM

Quote:

Originally Posted by montylee (Post 3223817)
My current code is using UDP. I can't change it to TCP as it's client requirement.

Almost everyone starts their first sockets project using UDP and then they discover that they have to implement most of the features of TCP (sequence numbers, retry mechanicms, etc.) to get it working. Some fortunate souls eventually give up and rewrite the entire thing using TCP. Reading the Stevens book saved me that heartache.

Quote:

Originally Posted by montylee (Post 3223817)
@David, i am splitting the files into chunks of 512 bytes and transferring them thru UDP socket. Should i try it with smaller size chunks? Should i post the code here? Please suggest...

You should definitely consider jiml8's suggestions about adding sequence numbers and checksums. That is exactly how UFTP works. I ported UFTP to a project that ran on both desktops and embedded systems. Although it works very well on a fast machine, it really got bogged down when it had to share the CPU on the embedded target.

Quote:

Originally Posted by montylee (Post 3223817)
I read a bit about expect and it is an excellent way of using SCP, but i have one doubt about the same:

I'll create an expect script and call it from my C code. But can i get some return value from the script so that i can get an idea whether the file transfer succeeded or failed? That's my only doubt.

Read the man pages for the "system" function. You can use the WEXITSTATUS macro to parse the return code of the script. I use that method to modify network settings using code like this
Code:

sprintf(cBuffer, "ifconfig %s %s", pszInterface, pszAddress);
iResult = system(cBuffer);
if (-1 == iResult)
{
    // Look at errno to find out what happened
}
else
{
    iResult = WEXITSTATUS(iResult);
    // iResult should have the return code of the script
}

Quote:

Originally Posted by montylee (Post 3223817)
I also thought about using NFS. I can simply mount the server directory containing my data on the clients and access the file directly. This seems to be a good solution.

NFS, Samba, or SSHFS. Six of one, half-dozen of another, etc.

Quote:

Originally Posted by montylee (Post 3223817)
Even if NFS or SCP works for me i would like to resolve the file transfer problem i am facing thru sockets.

We cannot really debug we cannot see. However if you are really trying to get help with a homework assignment it will be painfully obvious when we see your code.

montylee 07-23-2008 01:49 PM

thanks for the reply, specially for the sample code of getting the return value from the script.

I donno what's there with this homework assignment thing. Almost in every thread in this forum, ppl keep asking if it's an homework assignment. As i already told i am not a student. I am a software engineer working on a client's project.

I have a simple function for file transfer. I'll isolate the function and post the code here tomorrow.

For now, i think i am going with NFS mount for sharing files but as i mentioned i would like to resolve the socket problem as well even if i won't use it my code.

Mr. C. 07-23-2008 02:02 PM

Please try to understand the number of student homework assignments posted here is great; there are a lot of lazy, loser students who try to fool helpers into doing their assignments. Sometimes they succeed, often they are so stupid that they are easily detected.

Your assignment and requirements seem artificially and pointlessly constrained; clients usually specify end requirements and goals not implementation specifics as you have presented. And re-implementing the wheel is another oddity.

chrism01 07-23-2008 07:54 PM

Nicely put, esp the 2nd para. We've had this sort of qn (client insists on UDP for txfr) before...
As I said to that one, I'd strongly suggest explaining (politely/professionally) to the client why that's a bad idea.

montylee 07-24-2008 12:52 AM

Actually the client's entire architecture uses UDP for data transfer as they provide movies and games on demand. And for transferring such huge data UDP is the de facto. I have to design a tool for them, so i have to use UDP. I can't ask them to change their entire architecture for the sake of implementing a small tool which i am developing.

And by the way, requirements for this project were not done by me, so using UDP was already decided, so i have to stick with it.

If you still don't believe me, i can't do anything. I'll probably go with NFS mount and if the client approves it, it's good. Otherwise i'll go with SCP using expect.

Thanks for helping me out for this problem. Special thanks for @David1357 for posting sample codes.

If anybody believes that i am not a student, then i paste the sample UDP transfer code i am using. I'll try debugging it a bit by varying the MTU size.

montylee 07-24-2008 01:37 AM

lol, i had made some changes in the code. I tried transferring a file contains 8000+ lines and it was transferred correctly. Don't know how it is working properly now.

I am using a wireless network and transferring files between a laptop and a PC.

Here's the code being used for sending the file:
Code:

#define PACKET_SIZE        512
/* Open the file to be sent to the server */
if ((fd = open(deviceInfo->recFile, O_RDONLY)) < 0) {
        sprintf (message, "Error opening file '%s': %s", deviceInfo->recFile, strerror (errno));
        send_msg_to_server (message);
        return NULL;
}
/* Send the file contents to the automation server */
while ((numbytes = read (fd, buf, PACKET_SIZE - 1)) > 0) {
        buf[numbytes] = '\0';
        if ((numbytes = send_msg_to_server (buf)) == -1) {
                close (fd);
                return NULL;
        }
}
close (fd);

Here's the code for receiving the file:
Code:

#define PACKET_SIZE        512
ufds[0].fd = g_sockfd;
ufds[0].events = POLLIN;
addr_len = sizeof (recv_addr);
/* Read the file contents from the socket and write it to a text file */
while ((retval = poll (ufds, 1, TIMEOUT)) > 0) {
        if ((numbytes = recvfrom (g_sockfd, buf, PACKET_SIZE, 0,
                (struct sockaddr *) &recv_addr, &addr_len)) > 0) {
                buf[numbytes] = '\0';
                if ((numbytes = fwrite(&buf, strlen (buf), 1, fpRec)) <= 0) {
                        perror("fwrite");
                        return -1;
                }
                fflush(fpRec);
        } else {
                perror("recvfrom");
                return -1;
        }
}

In the above code send_msg_to_server() API just uses sendto() function to send the argument string to the server.


I tested the above code with a large file and it worked correctly. Earlier the same code was giving me problems.

I had tested the code when it was giving problems. I used a simply counter variable to count the number of times the sendto() and recvfrom() commands were being called. In case of error (large file), the counter variable was different for sendto() and recvfrom() but now it's same, so it's working properly now. In case of error, the counter variable in sendto() was equal to the number of lines in the file, but it was less in the recvfrom() function. So, i guess that the sendto() was sending the bytes correctly, but some bytes were getting lost, so recvfrom() wasn't getting the entire data.

Let me know if my code is correct.

Mr. C. 07-24-2008 02:03 AM

Last I looked, scp was not UDP-based, so that breaks your basic UDP requirement.

And NFS??? That just plain makes no sense. How are you going to handle security, hung mounts, excessive retransmits, user and group ids, and generally lackluster performance? I don't think you know what you're getting into here.

If you want performance, blast the bits yourself as others have mentioned, and handle file integrity as necessary. Don't count on bytes transmitted vs. bytes received as an integrity check.

David1357 07-25-2008 07:59 AM

Quote:

Originally Posted by montylee (Post 3224546)
lol, i had made some changes in the code. I tried transferring a file contains 8000+ lines and it was transferred correctly. Don't know how it is working properly now.

Do you use any kind of version tracking system? If so, you can diff your current version against your previous version. If you are not using a version tracking system, you have just found out why you need to start.

Quote:

Originally Posted by montylee (Post 3224546)
Here's the code being used for sending the file:
Code:

#define PACKET_SIZE        512
/* Open the file to be sent to the server */
if ((fd = open(deviceInfo->recFile, O_RDONLY)) < 0) {
        sprintf (message, "Error opening file '%s': %s", deviceInfo->recFile, strerror (errno));
        send_msg_to_server (message);
        return NULL;
}
/* Send the file contents to the automation server */
while ((numbytes = read (fd, buf, PACKET_SIZE - 1)) > 0) {
        buf[numbytes] = '\0';
        if ((numbytes = send_msg_to_server (buf)) == -1) {
                close (fd);
                return NULL;
        }
}
close (fd);


I would have used a do/while loop here which would allow you to check the return value of numbytes in a separate statement. On Linux, the return value for an error is -1, but if numbytes is unsigned, it will never be "< 0". Splitting out the check of the return value into separate statements can help you find bugs like that.

Quote:

Originally Posted by montylee (Post 3224546)
Here's the code for receiving the file:
Code:

#define PACKET_SIZE        512
ufds[0].fd = g_sockfd;
ufds[0].events = POLLIN;
addr_len = sizeof (recv_addr);
/* Read the file contents from the socket and write it to a text file */
while ((retval = poll (ufds, 1, TIMEOUT)) > 0) {
        if ((numbytes = recvfrom (g_sockfd, buf, PACKET_SIZE, 0,
                (struct sockaddr *) &recv_addr, &addr_len)) > 0) {
                buf[numbytes] = '\0';
                if ((numbytes = fwrite(&buf, strlen (buf), 1, fpRec)) <= 0) {
                        perror("fwrite");
                        return -1;
                }
                fflush(fpRec);
        } else {
                perror("recvfrom");
                return -1;
        }
}


Well, this is exactly what I was warning you about. You have no sanity check on your file chunks. Usually, people create header structures like this
Code:

typedef struct _CHUNK_HEADER
{
    int    iType;        // Allows for sending different chunk types
    int    iSize;        // Allows for validating the size
    int    iChecksum;    // Allows for detection of corrupt chunks
    int    iSequence;    // Allows for detection of missing chunks
} CHUNK_HEADER, *PCHUNK_HEADER __attribute__((packed));

Without some sort of header like this on each chunk, you have no idea whether what you received is what you sent. I have had people tell me that the checksum is unnecessary, but I have seen cases where the IP checksum passed because it does not handle bursts of errors very well. The probability of both checksums failing is very low.

Quote:

Originally Posted by montylee (Post 3224546)
So, i guess that the sendto() was sending the bytes correctly, but some bytes were getting lost, so recvfrom() wasn't getting the entire data.

If you were using sequence numbers, you would know if you were losing chunks. You really need to beef up your error checking and your "protocol" if you want to figure out why things work intermittently.

montylee 07-25-2008 08:52 AM

Quote:

Originally Posted by Mr. C. (Post 3224564)
Last I looked, scp was not UDP-based, so that breaks your basic UDP requirement.

And NFS??? That just plain makes no sense. How are you going to handle security, hung mounts, excessive retransmits, user and group ids, and generally lackluster performance? I don't think you know what you're getting into here.

If you want performance, blast the bits yourself as others have mentioned, and handle file integrity as necessary. Don't count on bytes transmitted vs. bytes received as an integrity check.

Thanks for the suggestion! I'll look into what solution i can use.


Quote:

Originally Posted by David1357 (Post 3226008)
Do you use any kind of version tracking system? If so, you can diff your current version against your previous version. If you are not using a version tracking system, you have just found out why you need to start.

I use Serena Version Manager. Actually i have made numerous changes to the code but no change to the file sending code.



Quote:

Originally Posted by David1357 (Post 3226008)
I would have used a do/while loop here which would allow you to check the return value of numbytes in a separate statement. On Linux, the return value for an error is -1, but if numbytes is unsigned, it will never be "< 0". Splitting out the check of the return value into separate statements can help you find bugs like that.

Thanks for the suggestion. I'll look into this.


Quote:

Originally Posted by David1357 (Post 3226008)
Well, this is exactly what I was warning you about. You have no sanity check on your file chunks. Usually, people create header structures like this
Code:

typedef struct _CHUNK_HEADER
{
    int    iType;        // Allows for sending different chunk types
    int    iSize;        // Allows for validating the size
    int    iChecksum;    // Allows for detection of corrupt chunks
    int    iSequence;    // Allows for detection of missing chunks
} CHUNK_HEADER, *PCHUNK_HEADER __attribute__((packed));

Without some sort of header like this on each chunk, you have no idea whether what you received is what you sent. I have had people tell me that the checksum is unnecessary, but I have seen cases where the IP checksum passed because it does not handle bursts of errors very well. The probability of both checksums failing is very low.



If you were using sequence numbers, you would know if you were losing chunks. You really need to beef up your error checking and your "protocol" if you want to figure out why things work intermittently.

Can u please point me to a good tutorial where i can find info about headers and such stuff.

I referred Beej's networking guide and i only know that i can send a string buffer through socket using sendto(). I have no idea about associating headers with the message and decoding it at the receiving end. A link to a tutorial would be of great help.

David1357 07-25-2008 04:39 PM

Quote:

Originally Posted by montylee (Post 3226050)
Can u please point me to a good tutorial where i can find info about headers and such stuff.

You make your own headers:
Code:

// Use static so that you don't waste the stack
static unsigned char ucBuffer[sizeof(CHUNK_HEADER) + 512];
PCHUNK_HEADER *pHeader = (PCHUNK_HEADER) ucBuffer;

pHeader->iType = FILE_CHUNK;
pHeader->iSize = sizeof(ucBuffer);
for (i = 0; i < iFileLength; i++)
{
    fread(&ucBuffer[sizeof(CHUNK_HEADER)], 512, 1, fd);
    CalculateChecksum(ucBuffer, &pHeader->iSize);
    pHeader->iSequence = i;
    write(sd, ucBuffer, sizeof(ucBuffer));
}

Obviously, this is just skeleton code and I have left out error checking and a lot of the details of handling files and sockets. But this should give you the basic idea of how to add your own headers to the data you send. If it does not, you might want to invest in a good book.

montylee 07-26-2008 02:58 AM

thanks for the sample code David!
Can u please suggest some online links or books where i can find more information regarding this.

chrism01 07-27-2008 07:50 PM

If you don't want to do your own checksum calcn, invoke md5sum (man md5sum) and calc that at the sender, send the checksum value as part of the msg eg in the 'header' referred to, then re-calc on the receiver and compare to the sent value.

montylee 07-28-2008 10:45 AM

actually i am already using md5sum but currently i am using it on the entire file. I'll use it per message now.

Mr. C. 07-28-2008 12:10 PM

There are plenty of open source checksum algorithms that will allow you to perform checksums on chunks. This will allow you to checksum each chunk before dumping it on the wire, and upon receiving it. This may allow you to provide faster failure detection and recovery.

David1357 08-11-2008 07:20 AM

Quote:

Originally Posted by montylee (Post 3226681)
thanks for the sample code David!
Can u please suggest some online links or books where i can find more information regarding this.

I have never seen code like that in a book. You will find code similar to that in the Linux kernel source. If you do find a book with sample code like that in it, please post the title here for everyone's benefit.


All times are GMT -5. The time now is 08:45 AM.