Best way to transfer files in Linux thru C programming?
ProgrammingThis forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
Best way to transfer files in Linux thru C programming?
Here's my requirement:
I have a server on which some files are present. There are N number of clients. The server sends a command to the clients with a name of a file. The clients need to retrieve the file from the server.
Currently i am just using sockets to send the file contents from the server to client, but this method has a problem. When i transfer a large file, the client doesn't receive the entire file contents i.e. some file contents are lost. The method works for small files though. All my code is in ANSI C.
Now, what is the best possible method to get the files from the server? The server might send different file names to different clients, so each client needs to retrieve the respective file from the server.
I am thinking of using SCP or NFS for file transfer. I need to use SCP or NFS from within my C code, so if the server requires a password for SCP, i need to automate this in my code. For e.g. the user can specify the user name and password in a config file at the client side. i'll just read the config file, get the user name, password from the file and use SCP to get the files from the server.
There are no major security requirements but i think the clients should specify a password before it can get the file from the server.
Basically i need to do this thru my C program. how can i do that? I think i can use SCP or TFTP.
I was thinking of using cURL but i am not sure if i should use it. The clients in my case are embedded machines with limited memory.
You started out developing a sockets-based networking application, but didn't get it to work correctly. So you're scrapping that approach, and are now thinking you should invoke an existing transfer application from within your own?
I don't get it - what's the point now of wrapping an existing file transfer application into another layer?
Could be... a homework assignment you couldn't complete?
lol i am not a student anymore I am working on a client project in Linux.
That TFTP was mentioned by someone on another forum, so i just copy pasted my thread from that forum. I can give u the link to that forum if you want. I'll edit my post...
Actually my application is still using sockets for other communication stuff. I want to use something else just for file transfer as using sockets is giving me a problem. I am not scrapping sockets or anything. I wanted to know what method i can use for file transfer in a C program.
Currently i am just using sockets to send the file contents from the server to client, but this method has a problem. When i transfer a large file, the client doesn't receive the entire file contents i.e. some file contents are lost. The method works for small files though. All my code is in ANSI C.
I have written code to transfer files in C many times. Are you using UDP or TCP? Are you trying to send the file all at once or are you splitting it up into pieces? The problem is solvable, but depending on your code, it may take a while to debug.
Quote:
Originally Posted by montylee
I am thinking of using SCP or NFS for file transfer. I need to use SCP or NFS from within my C code, so if the server requires a password for SCP, i need to automate this in my code.
The only way I have been able to successfully use SCP in a script without shared keys is using expect. For Ubuntu, it can be easily installed using apt or aptitude.
Here is the script I used (Note: The location of the brackets seems to be important)
Or you can open a pty and fork/exec the transfer app as if it were on a terminal. Still, the method is just plain silly; there are so many other dependencies. To each his own.
How about taking an existing file transfer protocol, and embed the code into you application? This way you can use an existing known-good tool on each end of the transfer while developing and debugging the peer file transfer code. There are many options, all of which should have readily available code and documentation that describes the protocol. Many protocols were developed to run on simple machinery, and are not complex to implement.
My sense is that your existing code can probably be debugged and made to work without a huge effort, especially if you are using a known-good TCP stack, and are transferring data using TCP. If the relevant code segment is not huge, perhaps posting it here will let a keen eye spot the problem.
--- rod.
thanks for the replies
My current code is using UDP. I can't change it to TCP as it's client requirement.
The file i want to transfer has the following format:
id=1,type=1,code=2,value=1,tv_sec=2,tv_usec=222
My file contains multiple lines of the above type. When the number of lines is small, the file is transferred properly but when the number of lines increase, some lines are corrupted or not transferred. Even on a single machine i got this problem.
@David, i am splitting the files into chunks of 512 bytes and transferring them thru UDP socket. Should i try it with smaller size chunks? Should i post the code here? Please suggest...
I read a bit about expect and it is an excellent way of using SCP, but i have one doubt about the same:
I'll create an expect script and call it from my C code. But can i get some return value from the script so that i can get an idea whether the file transfer succeeded or failed? That's my only doubt.
I also thought about using NFS. I can simply mount the server directory containing my data on the clients and access the file directly. This seems to be a good solution.
What do you guys suggest?
Even if NFS or SCP works for me i would like to resolve the file transfer problem i am facing thru sockets.
UDP is not guaranteed delivery, but on a single machine you certainly shouldn't be having trouble with it. I would guess you have a bad pointer someplace.
To deploy a reliable protocol using UDP over the internet it seems to me you'll have to break your transfer up into packets and tag each packet with a number and a checksum. Also maybe some header with a total number of packets. Then, if any packets arrive corrupted or do not arrive at all, the recipient can request a resend.
Or you could zip the whole thing up and send it via ftp.
I have a commercial app, that I sell, which does pretty much what you want. I use UDP because it is stateless and connectionless and that is how I want it. I have found it to be very advantageous to zip up large text files before sending them, and unzip them at the other end. Faster that way, and less likely to lose a packet that has to be resent.
My current code is using UDP. I can't change it to TCP as it's client requirement.
Almost everyone starts their first sockets project using UDP and then they discover that they have to implement most of the features of TCP (sequence numbers, retry mechanicms, etc.) to get it working. Some fortunate souls eventually give up and rewrite the entire thing using TCP. Reading the Stevens book saved me that heartache.
Quote:
Originally Posted by montylee
@David, i am splitting the files into chunks of 512 bytes and transferring them thru UDP socket. Should i try it with smaller size chunks? Should i post the code here? Please suggest...
You should definitely consider jiml8's suggestions about adding sequence numbers and checksums. That is exactly how UFTP works. I ported UFTP to a project that ran on both desktops and embedded systems. Although it works very well on a fast machine, it really got bogged down when it had to share the CPU on the embedded target.
Quote:
Originally Posted by montylee
I read a bit about expect and it is an excellent way of using SCP, but i have one doubt about the same:
I'll create an expect script and call it from my C code. But can i get some return value from the script so that i can get an idea whether the file transfer succeeded or failed? That's my only doubt.
Read the man pages for the "system" function. You can use the WEXITSTATUS macro to parse the return code of the script. I use that method to modify network settings using code like this
Code:
sprintf(cBuffer, "ifconfig %s %s", pszInterface, pszAddress);
iResult = system(cBuffer);
if (-1 == iResult)
{
// Look at errno to find out what happened
}
else
{
iResult = WEXITSTATUS(iResult);
// iResult should have the return code of the script
}
Quote:
Originally Posted by montylee
I also thought about using NFS. I can simply mount the server directory containing my data on the clients and access the file directly. This seems to be a good solution.
NFS, Samba, or SSHFS. Six of one, half-dozen of another, etc.
Quote:
Originally Posted by montylee
Even if NFS or SCP works for me i would like to resolve the file transfer problem i am facing thru sockets.
We cannot really debug we cannot see. However if you are really trying to get help with a homework assignment it will be painfully obvious when we see your code.
thanks for the reply, specially for the sample code of getting the return value from the script.
I donno what's there with this homework assignment thing. Almost in every thread in this forum, ppl keep asking if it's an homework assignment. As i already told i am not a student. I am a software engineer working on a client's project.
I have a simple function for file transfer. I'll isolate the function and post the code here tomorrow.
For now, i think i am going with NFS mount for sharing files but as i mentioned i would like to resolve the socket problem as well even if i won't use it my code.
Please try to understand the number of student homework assignments posted here is great; there are a lot of lazy, loser students who try to fool helpers into doing their assignments. Sometimes they succeed, often they are so stupid that they are easily detected.
Your assignment and requirements seem artificially and pointlessly constrained; clients usually specify end requirements and goals not implementation specifics as you have presented. And re-implementing the wheel is another oddity.
Nicely put, esp the 2nd para. We've had this sort of qn (client insists on UDP for txfr) before...
As I said to that one, I'd strongly suggest explaining (politely/professionally) to the client why that's a bad idea.
Actually the client's entire architecture uses UDP for data transfer as they provide movies and games on demand. And for transferring such huge data UDP is the de facto. I have to design a tool for them, so i have to use UDP. I can't ask them to change their entire architecture for the sake of implementing a small tool which i am developing.
And by the way, requirements for this project were not done by me, so using UDP was already decided, so i have to stick with it.
If you still don't believe me, i can't do anything. I'll probably go with NFS mount and if the client approves it, it's good. Otherwise i'll go with SCP using expect.
Thanks for helping me out for this problem. Special thanks for @David1357 for posting sample codes.
If anybody believes that i am not a student, then i paste the sample UDP transfer code i am using. I'll try debugging it a bit by varying the MTU size.
lol, i had made some changes in the code. I tried transferring a file contains 8000+ lines and it was transferred correctly. Don't know how it is working properly now.
I am using a wireless network and transferring files between a laptop and a PC.
Here's the code being used for sending the file:
Code:
#define PACKET_SIZE 512
/* Open the file to be sent to the server */
if ((fd = open(deviceInfo->recFile, O_RDONLY)) < 0) {
sprintf (message, "Error opening file '%s': %s", deviceInfo->recFile, strerror (errno));
send_msg_to_server (message);
return NULL;
}
/* Send the file contents to the automation server */
while ((numbytes = read (fd, buf, PACKET_SIZE - 1)) > 0) {
buf[numbytes] = '\0';
if ((numbytes = send_msg_to_server (buf)) == -1) {
close (fd);
return NULL;
}
}
close (fd);
Here's the code for receiving the file:
Code:
#define PACKET_SIZE 512
ufds[0].fd = g_sockfd;
ufds[0].events = POLLIN;
addr_len = sizeof (recv_addr);
/* Read the file contents from the socket and write it to a text file */
while ((retval = poll (ufds, 1, TIMEOUT)) > 0) {
if ((numbytes = recvfrom (g_sockfd, buf, PACKET_SIZE, 0,
(struct sockaddr *) &recv_addr, &addr_len)) > 0) {
buf[numbytes] = '\0';
if ((numbytes = fwrite(&buf, strlen (buf), 1, fpRec)) <= 0) {
perror("fwrite");
return -1;
}
fflush(fpRec);
} else {
perror("recvfrom");
return -1;
}
}
In the above code send_msg_to_server() API just uses sendto() function to send the argument string to the server.
I tested the above code with a large file and it worked correctly. Earlier the same code was giving me problems.
I had tested the code when it was giving problems. I used a simply counter variable to count the number of times the sendto() and recvfrom() commands were being called. In case of error (large file), the counter variable was different for sendto() and recvfrom() but now it's same, so it's working properly now. In case of error, the counter variable in sendto() was equal to the number of lines in the file, but it was less in the recvfrom() function. So, i guess that the sendto() was sending the bytes correctly, but some bytes were getting lost, so recvfrom() wasn't getting the entire data.
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.