Best way to transfer files in Linux thru C programming?
Here's my requirement:
I have a server on which some files are present. There are N number of clients. The server sends a command to the clients with a name of a file. The clients need to retrieve the file from the server. Currently i am just using sockets to send the file contents from the server to client, but this method has a problem. When i transfer a large file, the client doesn't receive the entire file contents i.e. some file contents are lost. The method works for small files though. All my code is in ANSI C. Now, what is the best possible method to get the files from the server? The server might send different file names to different clients, so each client needs to retrieve the respective file from the server. I am thinking of using SCP or NFS for file transfer. I need to use SCP or NFS from within my C code, so if the server requires a password for SCP, i need to automate this in my code. For e.g. the user can specify the user name and password in a config file at the client side. i'll just read the config file, get the user name, password from the file and use SCP to get the files from the server. There are no major security requirements but i think the clients should specify a password before it can get the file from the server. Basically i need to do this thru my C program. how can i do that? I think i can use SCP or TFTP. I was thinking of using cURL but i am not sure if i should use it. The clients in my case are embedded machines with limited memory. Please help me. |
You started out developing a sockets-based networking application, but didn't get it to work correctly. So you're scrapping that approach, and are now thinking you should invoke an existing transfer application from within your own?
I don't get it - what's the point now of wrapping an existing file transfer application into another layer? Could be... a homework assignment you couldn't complete? Quote:
|
lol i am not a student anymore :) I am working on a client project in Linux.
That TFTP was mentioned by someone on another forum, so i just copy pasted my thread from that forum. I can give u the link to that forum if you want. I'll edit my post... Actually my application is still using sockets for other communication stuff. I want to use something else just for file transfer as using sockets is giving me a problem. I am not scrapping sockets or anything. I wanted to know what method i can use for file transfer in a C program. |
Quote:
Quote:
Here is the script I used (Note: The location of the brackets seems to be important) It took some digging around on the web and some local tinkering to get the magic right. If you are really space limited and security is not crucial, it may be easier to use FTP. It can be driven directly from a bash script. |
Or you can open a pty and fork/exec the transfer app as if it were on a terminal. Still, the method is just plain silly; there are so many other dependencies. To each his own.
|
How about taking an existing file transfer protocol, and embed the code into you application? This way you can use an existing known-good tool on each end of the transfer while developing and debugging the peer file transfer code. There are many options, all of which should have readily available code and documentation that describes the protocol. Many protocols were developed to run on simple machinery, and are not complex to implement.
My sense is that your existing code can probably be debugged and made to work without a huge effort, especially if you are using a known-good TCP stack, and are transferring data using TCP. If the relevant code segment is not huge, perhaps posting it here will let a keen eye spot the problem. --- rod. |
Makes sense to debug your socket code.
All the higher level solns boil down to a socket (TCP or UDP) connection anyway... http://en.wikipedia.org/wiki/Internet_layer |
thanks for the replies :)
My current code is using UDP. I can't change it to TCP as it's client requirement. The file i want to transfer has the following format: id=1,type=1,code=2,value=1,tv_sec=2,tv_usec=222 My file contains multiple lines of the above type. When the number of lines is small, the file is transferred properly but when the number of lines increase, some lines are corrupted or not transferred. Even on a single machine i got this problem. @David, i am splitting the files into chunks of 512 bytes and transferring them thru UDP socket. Should i try it with smaller size chunks? Should i post the code here? Please suggest... I read a bit about expect and it is an excellent way of using SCP, but i have one doubt about the same: I'll create an expect script and call it from my C code. But can i get some return value from the script so that i can get an idea whether the file transfer succeeded or failed? That's my only doubt. I also thought about using NFS. I can simply mount the server directory containing my data on the clients and access the file directly. This seems to be a good solution. What do you guys suggest? Even if NFS or SCP works for me i would like to resolve the file transfer problem i am facing thru sockets. |
UDP is not guaranteed delivery, but on a single machine you certainly shouldn't be having trouble with it. I would guess you have a bad pointer someplace.
To deploy a reliable protocol using UDP over the internet it seems to me you'll have to break your transfer up into packets and tag each packet with a number and a checksum. Also maybe some header with a total number of packets. Then, if any packets arrive corrupted or do not arrive at all, the recipient can request a resend. Or you could zip the whole thing up and send it via ftp. I have a commercial app, that I sell, which does pretty much what you want. I use UDP because it is stateless and connectionless and that is how I want it. I have found it to be very advantageous to zip up large text files before sending them, and unzip them at the other end. Faster that way, and less likely to lose a packet that has to be resent. |
Quote:
Quote:
Quote:
Quote:
Quote:
|
thanks for the reply, specially for the sample code of getting the return value from the script.
I donno what's there with this homework assignment thing. Almost in every thread in this forum, ppl keep asking if it's an homework assignment. As i already told i am not a student. I am a software engineer working on a client's project. I have a simple function for file transfer. I'll isolate the function and post the code here tomorrow. For now, i think i am going with NFS mount for sharing files but as i mentioned i would like to resolve the socket problem as well even if i won't use it my code. |
Please try to understand the number of student homework assignments posted here is great; there are a lot of lazy, loser students who try to fool helpers into doing their assignments. Sometimes they succeed, often they are so stupid that they are easily detected.
Your assignment and requirements seem artificially and pointlessly constrained; clients usually specify end requirements and goals not implementation specifics as you have presented. And re-implementing the wheel is another oddity. |
Nicely put, esp the 2nd para. We've had this sort of qn (client insists on UDP for txfr) before...
As I said to that one, I'd strongly suggest explaining (politely/professionally) to the client why that's a bad idea. |
Actually the client's entire architecture uses UDP for data transfer as they provide movies and games on demand. And for transferring such huge data UDP is the de facto. I have to design a tool for them, so i have to use UDP. I can't ask them to change their entire architecture for the sake of implementing a small tool which i am developing.
And by the way, requirements for this project were not done by me, so using UDP was already decided, so i have to stick with it. If you still don't believe me, i can't do anything. I'll probably go with NFS mount and if the client approves it, it's good. Otherwise i'll go with SCP using expect. Thanks for helping me out for this problem. Special thanks for @David1357 for posting sample codes. If anybody believes that i am not a student, then i paste the sample UDP transfer code i am using. I'll try debugging it a bit by varying the MTU size. |
lol, i had made some changes in the code. I tried transferring a file contains 8000+ lines and it was transferred correctly. Don't know how it is working properly now.
I am using a wireless network and transferring files between a laptop and a PC. Here's the code being used for sending the file: Code:
#define PACKET_SIZE 512 Code:
#define PACKET_SIZE 512 I tested the above code with a large file and it worked correctly. Earlier the same code was giving me problems. I had tested the code when it was giving problems. I used a simply counter variable to count the number of times the sendto() and recvfrom() commands were being called. In case of error (large file), the counter variable was different for sendto() and recvfrom() but now it's same, so it's working properly now. In case of error, the counter variable in sendto() was equal to the number of lines in the file, but it was less in the recvfrom() function. So, i guess that the sendto() was sending the bytes correctly, but some bytes were getting lost, so recvfrom() wasn't getting the entire data. Let me know if my code is correct. |
Last I looked, scp was not UDP-based, so that breaks your basic UDP requirement.
And NFS??? That just plain makes no sense. How are you going to handle security, hung mounts, excessive retransmits, user and group ids, and generally lackluster performance? I don't think you know what you're getting into here. If you want performance, blast the bits yourself as others have mentioned, and handle file integrity as necessary. Don't count on bytes transmitted vs. bytes received as an integrity check. |
Quote:
Quote:
Quote:
Without some sort of header like this on each chunk, you have no idea whether what you received is what you sent. I have had people tell me that the checksum is unnecessary, but I have seen cases where the IP checksum passed because it does not handle bursts of errors very well. The probability of both checksums failing is very low. Quote:
|
Quote:
Quote:
Quote:
Quote:
I referred Beej's networking guide and i only know that i can send a string buffer through socket using sendto(). I have no idea about associating headers with the message and decoding it at the receiving end. A link to a tutorial would be of great help. |
Quote:
Obviously, this is just skeleton code and I have left out error checking and a lot of the details of handling files and sockets. But this should give you the basic idea of how to add your own headers to the data you send. If it does not, you might want to invest in a good book. |
thanks for the sample code David!
Can u please suggest some online links or books where i can find more information regarding this. |
If you don't want to do your own checksum calcn, invoke md5sum (man md5sum) and calc that at the sender, send the checksum value as part of the msg eg in the 'header' referred to, then re-calc on the receiver and compare to the sent value.
|
actually i am already using md5sum but currently i am using it on the entire file. I'll use it per message now.
|
There are plenty of open source checksum algorithms that will allow you to perform checksums on chunks. This will allow you to checksum each chunk before dumping it on the wire, and upon receiving it. This may allow you to provide faster failure detection and recovery.
|
Quote:
|
All times are GMT -5. The time now is 08:45 AM. |