Best way to transfer files in Linux thru C programming?
 

Here's my requirement:

I have a server on which some files are present. There are N number of clients. The server sends a command to the clients with a name of a file. The clients need to retrieve the file from the server.

Currently i am just using sockets to send the file contents from the server to client, but this method has a problem. When i transfer a large file, the client doesn't receive the entire file contents i.e. some file contents are lost. The method works for small files though. All my code is in ANSI C.

Now, what is the best possible method to get the files from the server? The server might send different file names to different clients, so each client needs to retrieve the respective file from the server.

I am thinking of using SCP or NFS for file transfer. I need to use SCP or NFS from within my C code, so if the server requires a password for SCP, i need to automate this in my code. For e.g. the user can specify the user name and password in a config file at the client side. i'll just read the config file, get the user name, password from the file and use SCP to get the files from the server.

There are no major security requirements but i think the clients should specify a password before it can get the file from the server.

Basically i need to do this thru my C program. how can i do that? I think i can use SCP or TFTP.

I was thinking of using cURL but i am not sure if i should use it. The clients in my case are embedded machines with limited memory.

Please help me.

You started out developing a sockets-based networking application, but didn't get it to work correctly. So you're scrapping that approach, and are now thinking you should invoke an existing transfer application from within your own?

I don't get it - what's the point now of wrapping an existing file transfer application into another layer?

Could be... a homework assignment you couldn't complete?

Like who mentioned? Your instructor?

lol i am not a student anymore :) I am working on a client project in Linux.
That TFTP was mentioned by someone on another forum, so i just copy pasted my thread from that forum. I can give u the link to that forum if you want. I'll edit my post...

Actually my application is still using sockets for other communication stuff. I want to use something else just for file transfer as using sockets is giving me a problem. I am not scrapping sockets or anything. I wanted to know what method i can use for file transfer in a C program.

I have written code to transfer files in C many times. Are you using UDP or TCP? Are you trying to send the file all at once or are you splitting it up into pieces? The problem is solvable, but depending on your code, it may take a while to debug.

The only way I have been able to successfully use SCP in a script without shared keys is using expect. For Ubuntu, it can be easily installed using apt or aptitude.

Here is the script I used (Note: The location of the brackets seems to be important)

Code:

---

#!/usr/bin/expect
spawn scp auser@amachine:~/testfile.txt .
expect "password:" {
        send "apassword\r"
        expect "100%"
}

---

It took some digging around on the web and some local tinkering to get the magic right.

If you are really space limited and security is not crucial, it may be easier to use FTP. It can be driven directly from a bash script.

Or you can open a pty and fork/exec the transfer app as if it were on a terminal. Still, the method is just plain silly; there are so many other dependencies. To each his own.

How about taking an existing file transfer protocol, and embed the code into you application? This way you can use an existing known-good tool on each end of the transfer while developing and debugging the peer file transfer code. There are many options, all of which should have readily available code and documentation that describes the protocol. Many protocols were developed to run on simple machinery, and are not complex to implement.
My sense is that your existing code can probably be debugged and made to work without a huge effort, especially if you are using a known-good TCP stack, and are transferring data using TCP. If the relevant code segment is not huge, perhaps posting it here will let a keen eye spot the problem.
--- rod.

Makes sense to debug your socket code.
All the higher level solns boil down to a socket (TCP or UDP) connection anyway...
http://en.wikipedia.org/wiki/Internet_layer

thanks for the replies :)
My current code is using UDP. I can't change it to TCP as it's client requirement.

The file i want to transfer has the following format:

id=1,type=1,code=2,value=1,tv_sec=2,tv_usec=222

My file contains multiple lines of the above type. When the number of lines is small, the file is transferred properly but when the number of lines increase, some lines are corrupted or not transferred. Even on a single machine i got this problem.

@David, i am splitting the files into chunks of 512 bytes and transferring them thru UDP socket. Should i try it with smaller size chunks? Should i post the code here? Please suggest...

I read a bit about expect and it is an excellent way of using SCP, but i have one doubt about the same:

I'll create an expect script and call it from my C code. But can i get some return value from the script so that i can get an idea whether the file transfer succeeded or failed? That's my only doubt.

I also thought about using NFS. I can simply mount the server directory containing my data on the clients and access the file directly. This seems to be a good solution.

What do you guys suggest?

Even if NFS or SCP works for me i would like to resolve the file transfer problem i am facing thru sockets.

UDP is not guaranteed delivery, but on a single machine you certainly shouldn't be having trouble with it. I would guess you have a bad pointer someplace.

To deploy a reliable protocol using UDP over the internet it seems to me you'll have to break your transfer up into packets and tag each packet with a number and a checksum. Also maybe some header with a total number of packets. Then, if any packets arrive corrupted or do not arrive at all, the recipient can request a resend.

Or you could zip the whole thing up and send it via ftp.

I have a commercial app, that I sell, which does pretty much what you want. I use UDP because it is stateless and connectionless and that is how I want it. I have found it to be very advantageous to zip up large text files before sending them, and unzip them at the other end. Faster that way, and less likely to lose a packet that has to be resent.

Almost everyone starts their first sockets project using UDP and then they discover that they have to implement most of the features of TCP (sequence numbers, retry mechanicms, etc.) to get it working. Some fortunate souls eventually give up and rewrite the entire thing using TCP. Reading the Stevens book saved me that heartache.

You should definitely consider jiml8's suggestions about adding sequence numbers and checksums. That is exactly how UFTP works. I ported UFTP to a project that ran on both desktops and embedded systems. Although it works very well on a fast machine, it really got bogged down when it had to share the CPU on the embedded target.

Read the man pages for the "system" function. You can use the WEXITSTATUS macro to parse the return code of the script. I use that method to modify network settings using code like this

Code:

---

sprintf(cBuffer, "ifconfig %s %s", pszInterface, pszAddress);
iResult = system(cBuffer);
if (-1 == iResult)
{
    // Look at errno to find out what happened
}
else
{
    iResult = WEXITSTATUS(iResult);
    // iResult should have the return code of the script
}

---

NFS, Samba, or SSHFS. Six of one, half-dozen of another, etc.

We cannot really debug we cannot see. However if you are really trying to get help with a homework assignment it will be painfully obvious when we see your code.

thanks for the reply, specially for the sample code of getting the return value from the script.

I donno what's there with this homework assignment thing. Almost in every thread in this forum, ppl keep asking if it's an homework assignment. As i already told i am not a student. I am a software engineer working on a client's project.

I have a simple function for file transfer. I'll isolate the function and post the code here tomorrow.

For now, i think i am going with NFS mount for sharing files but as i mentioned i would like to resolve the socket problem as well even if i won't use it my code.

Please try to understand the number of student homework assignments posted here is great; there are a lot of lazy, loser students who try to fool helpers into doing their assignments. Sometimes they succeed, often they are so stupid that they are easily detected.

Your assignment and requirements seem artificially and pointlessly constrained; clients usually specify end requirements and goals not implementation specifics as you have presented. And re-implementing the wheel is another oddity.

Nicely put, esp the 2nd para. We've had this sort of qn (client insists on UDP for txfr) before...
As I said to that one, I'd strongly suggest explaining (politely/professionally) to the client why that's a bad idea.

Actually the client's entire architecture uses UDP for data transfer as they provide movies and games on demand. And for transferring such huge data UDP is the de facto. I have to design a tool for them, so i have to use UDP. I can't ask them to change their entire architecture for the sake of implementing a small tool which i am developing.

And by the way, requirements for this project were not done by me, so using UDP was already decided, so i have to stick with it.

If you still don't believe me, i can't do anything. I'll probably go with NFS mount and if the client approves it, it's good. Otherwise i'll go with SCP using expect.

Thanks for helping me out for this problem. Special thanks for @David1357 for posting sample codes.

If anybody believes that i am not a student, then i paste the sample UDP transfer code i am using. I'll try debugging it a bit by varying the MTU size.

lol, i had made some changes in the code. I tried transferring a file contains 8000+ lines and it was transferred correctly. Don't know how it is working properly now.

I am using a wireless network and transferring files between a laptop and a PC.

Here's the code being used for sending the file:
Here's the code for receiving the file:
In the above code send_msg_to_server() API just uses sendto() function to send the argument string to the server.


I tested the above code with a large file and it worked correctly. Earlier the same code was giving me problems.

I had tested the code when it was giving problems. I used a simply counter variable to count the number of times the sendto() and recvfrom() commands were being called. In case of error (large file), the counter variable was different for sendto() and recvfrom() but now it's same, so it's working properly now. In case of error, the counter variable in sendto() was equal to the number of lines in the file, but it was less in the recvfrom() function. So, i guess that the sendto() was sending the bytes correctly, but some bytes were getting lost, so recvfrom() wasn't getting the entire data.

Let me know if my code is correct.