LinuxQuestions.org - Sending folder content over TCP socket

- Programming (https://www.linuxquestions.org/questions/programming-9/)

- - Sending folder content over TCP socket (https://www.linuxquestions.org/questions/programming-9/sending-folder-content-over-tcp-socket-4175431209/)

Sending folder content over TCP socket

Hi everyone...

I'm here because I've got some troubles on sending all the files in a specific folder, through TCP socket (network socket, non Unix socket).

I already know how to use "sendmsg", "recmsg", "send", "recv" or "write & read", to read and/or write from/to a socket.

So... what I need to do is just like what I'm going to explain here following...

For example...

The client process reads the content of a directory, where inside we got five files. I have to write a function in C language to "get" all this five files and "upload" everything to another machine using TCP sockets.

Of course, on server-side, there will be another function that gets all the stuff from client-side and "writes it down".

In few words, I shoud write a function to upload the content of a directory, to a server.

P.S.

I haven't got any troubles on sending strings, integers, chars, buffers and, of course, a single file at time: I already know how to upload, or exchange, a SINGLE file to a server.

Any suggestions ?

Thanks in advance.

So, to transmit all of the files in a directory, you will need to open the directory for reading (opendir() in C; you haven't specified a language), then iteratively readdir() each entry, and send it if it is a regular file. You will need to contrive some protocol that tells the receiving host about things like filenames, perhaps file ownership, permissions, etc, and when to quit listening. This can be done as part of a header in each chunk of data that you send, or any other way you see fit. It just has to be somehow distinct from the file data. If were me, I think I would be inclined to send the dirent structure returned by readdir() as the file meta-data.

--- rod.

This is the way I was thinking about but, it gets extremely complicated. However, a solution could be: read file name, send it to the server, the server creates a file with such name and gets data.

But my of my... there should be a simpler way

It doesn't sound too complicated to me. You've already done the hardest part; establishing the TCP connection and transferring a specified file. All that's left is iterating over a list of file names, and acquiring that list is little different from reading it from a file, which might make a reasonable first step to get the iteration part right.
Don't forget to divide and conquer. Get each little part working independently, and once you've got them all working, put them together to form the whole. It is reasonable to always try to simplify the process, however at some point you have to recognize where the information lies, and what the process is for your program to get it. There is just no getting around that part.

--- rod.

This will list all files in a dir

Code:

int main(int argc, char *argv[]) {

DIR *dp;

struct dirent *ep;



dp = opendir ("/home/amboxer21/Documents");

        if (dp != NULL) {

          while ((ep = readdir(dp)))

          puts (ep->d_name);

          (void) closedir (dp);

        }

                else

                perror ("Couldn't open the directory");



return 0;

}

Check out the doc here -> http://www.delorie.com/gnu/docs/glibc/libc_568.html C's glib library has some good functions/wrappers -> http://www.gtk.org/api/2.6/glib/glib...Utilities.html

I guess I can use "scadir(3)" to read a directory.

Anyway, what I'm trying to do is to use a struct that represents the file. Only two fields are required "file_title" and "file_data".

Code:

struct file

{

  char *file_title;



  char *file_data; 

};

It should be O.K. with padding and so on: I mean network byte ordering and related stuff.

I'll let you know and post some code.

This source code is pretty straightforward. However, I need to manage only text files: not binary files.

So this is what happens on client-side (no network stuff here)... Fire at will !

Code:

#include <stdio.h>

#include <stdlib.h>

#include <string.h>

#include <unistd.h>

#include <errno.h>

#include <dirent.h>



#ifndef _FILE_NAME_STR_MAX_LEN_

#define _FILE_NAME_STR_MAX_LEN_ 15

#endif /* _FILE_NAME_STR_MAX_LEN_ */



#ifndef _LOCAL_BUFFER_

#define _LOCAL_BUFFER_ 1024

#endif /* LOCAL_BUFFER_ */



/* "file" struct */

typedef struct

{

  char *file_name; /* name of current file */

  

  //char *file_data; /* data of current file */



} __current_file;



/* used to backup "errno" */

int err_num;



void set_data_and_name(char *dir_name);



/* ------------------------- */



int main(int argc, char **argv)

{

  system("clear");



  set_data_and_name(".");

  

  return 0;

}



/* ------------------------- */



void set_data_and_name(char *dir_name)

{

  int ret_val = 0;



  int files_num = 0;

  

  int i = 0;

  

  int file_name_str_len = 0;

      

  struct dirent **file_list;

  

  ret_val = scandir(dir_name, &file_list, 0, alphasort);

  

      if (ret_val < 0)

      {

        err_num = errno;

        

        (void)fprintf(stderr, "\"scandir\" error: %s\n", strerror(err_num));

        

        exit(EXIT_FAILURE);

      }

      

  /* skip "." and ".." */

  files_num = (ret_val - 2);



  __current_file this_file[files_num];



      /* set the structure */

      for (i = 2; i < ret_val; i++)

      {

        file_name_str_len = strlen(file_list[i]->d_name);

      

        this_file[i].file_name = (char *) malloc(file_name_str_len * sizeof(char *));

        

            if (this_file[i].file_name == NULL)

            {

              (void)fprintf(stdout, "Can't allocate memory... exiting !\n");

              

              exit(EXIT_FAILURE);

            }

      

        this_file[i].file_name = file_list[i]->d_name;

        

        (void)fprintf(stdout, "[ %d ]: %s\n", i, this_file[i].file_name);

      }

}

At this point I need to read each files and store the content in

Quote:

this_file[i].file_data

Why use C?
Why not just use scp

or how about using a shell script and netcat?

source:
tar cvf - *.blah | nc target_host 50666
target:
nc -l 50666 | tar xvf -

Quote:

Originally Posted by bigearsbilly (Post 4801067)

Why use C?
Why not just use scp

or how about using a shell script and netcat?

source:
tar cvf - *.blah | nc target_host 50666
target:
nc -l 50666 | tar xvf -

I don't use shell script and I use C language cuz (!(I like myself)) && (I hate myself): at least I guess.

Anyway, I'm from Italy... what about my "english". It sucks, right ?

At this point I have what follow here...

Code:

#include <stdio.h>

#include <stdlib.h>

#include <string.h>

#include <unistd.h>

#include <errno.h>

#include <dirent.h>

#include <sys/types.h>

#include <sys/stat.h>



#ifndef _FILE_NAME_STR_MAX_LEN_

#define _FILE_NAME_STR_MAX_LEN_ 15

#endif /* _FILE_NAME_STR_MAX_LEN_ */



#ifndef _LOCAL_BUFFER_

#define _LOCAL_BUFFER_ 1024

#endif /* LOCAL_BUFFER_ */



/* "file" struct */

typedef struct

{

  char *file_name; /* name of current file */

  

  char *file_data; /* data of current file */



} __current_file;



/* used to backup "errno" */

int err_num;



void set_data_and_name(char *dir_name);



char * read_file(char *file_name);



off_t get_file_size(char *file_name);



/* ------------------------- */



int main(int argc, char **argv)

{

  system("clear");



  set_data_and_name(".");

        

  return 0;

}



/* ------------------------- */



off_t get_file_size(char *file_name)

{

  int ret_val;



  struct stat sb;

  

  off_t file_size;

  

  ret_val = stat(file_name, &sb);

  

      if (ret_val < 0)

      {

        err_num = errno;

        

        (void)fprintf(stderr, "\"stat\" error: %s\n", strerror(err_num));

        

        exit(EXIT_FAILURE);

      }

        

  file_size = sb.st_size;

      

  return (file_size);

}



void set_data_and_name(char *dir_name)

{

  int ret_val = 0;



  int files_num = 0;

  

  int i = 0;

  

  struct dirent **file_list;

  

  off_t f_size;

  

  ret_val = scandir(dir_name, &file_list, 0, alphasort);

  

      if (ret_val < 0)

      {

        err_num = errno;

        

        (void)fprintf(stderr, "\"scandir\" error: %s\n", strerror(err_num));

        

        exit(EXIT_FAILURE);

      }

      

  /* skip "." and ".." */

  files_num = (ret_val - 2);



  __current_file this_file[files_num];



      /* set the structure */

      for (i = 2; i < ret_val; i++)

      {  

        this_file[i].file_name = (char *) malloc((int)file_list[i]->d_reclen  * sizeof(char *));

        

            if (this_file[i].file_name == NULL)

            {

              (void)fprintf(stdout, "Can't allocate memory... exiting !\n");

              

              exit(EXIT_FAILURE);

            }

      

        this_file[i].file_name = file_list[i]->d_name;

        

        f_size = get_file_size(this_file[i].file_name);

        

        this_file[i].file_data = (char *) malloc((long long)f_size * sizeof(char *));

        

            if (this_file[i].file_data == NULL)

            {

              (void)fprintf(stdout, "Can't allocate memory... exiting !\n");

              

              exit(EXIT_FAILURE);

            }

            

        this_file[i].file_data = read_file(this_file[i].file_name);  



        (void)fprintf(stdout, "%s\n\n", this_file[i].file_data);

      }

}





char *read_file(char *file_name)

{

  FILE *fp = NULL;

  

  long size = 0;

  

  char *local_buffer = NULL;

  

  size_t fread_result = 0;

  

  fp = fopen(file_name, "rb");

  

      if (fp == NULL)

      {

        err_num = errno;

        

        (void)fprintf(stderr, "\"fopen\" error: %s\n", strerror(err_num));

        

        exit(EXIT_FAILURE);

      }

      

  /* get file size */

  fseek(fp, 0, SEEK_END);

  size = ftell(fp);

  rewind(fp);

  

  local_buffer = (char *) malloc(size * sizeof(char *));

  

      if (local_buffer == NULL)

      {

        (void)fprintf(stderr, "Can't allocate memory... exiting !\n");

        

        exit(EXIT_FAILURE);

      }

      

  /* copy the file into the buffer */

  fread_result = fread(local_buffer, 1, size, fp);

  

      if (fread_result != size)

      {

        (void)fprintf(stderr, "Reading error... exiting !\n");

        

        exit(EXIT_FAILURE);

      }

      

  return (local_buffer);

  

  fclose(fp);

  

  free(local_buffer);

}

What I need now is to send that structure over a TCP socket: writing to the socket on client-side and reading from the socket on server-side. Am I right ?

A few comments:

Don't read the entire file, just to find out the size of the file. Use stat() (man 2 stat)
Don't try to read the entire file into memory. Choose a reasonable buffer size, and read + send iteratively until end-of-file.
It looks like you are using malloc to allocate storage for data, but only sending the struct holding pointers to the data. That won't work. Allocate storage statically in reasonably sized buffers. Populate the buffer iteratively, sending the data on each fill.

--- rod.

This is incredible: I'm not able to copy more than one file ! :cry:

Describe, in detail how you copy one file. How does the second and subsequent file fail to copy?

--- rod.

Quote:

Originally Posted by theNbomr (Post 4802540)

Describe, in detail how you copy one file. How does the second and subsequent file fail to copy?

--- rod.

To copy more than one file from A-side to B-side:

A-side: scan the directory and count the number of files. Send number of files to B-side.

B-side: get the number of files so that, from this side, we know how many files we need to open.

A-side: read each file's name and send, one at time, such string to B-side.

B-side: get each string: number of strings = number of files.

A-side: read file 1, fill buffer 1, send buffer 1

B-side: get buffer 1, write buffer 1.

and so on...

Is it right ? I really dunno how to do.

You should design your code to send one file, along with enough data for the peer side to know enough about the one file that it can create an entry for the file. At a minimum, this will be the name of the file.
The sender should do (pseudo-code):

Code:

open TCP connection



get file name to send



send file data (name, date, permissions, ownership, etc)

open file

while not end-of-file

  read some data

  send the data

done

close file



close TCP connection

You have already said you could send a single file. Use that code to send the single file. In post #5, amboxer21 gave you all the code you need to iterate over all of the files in a given directory. Put your code to send a single file inside the loop provided to you by amboxer21.
That is the basis of looping/iteration in computer programs. You make some code that does one thing. Use a loop to make that code run multiple times. Your code that does one thing will need to take parameters, so that it can operate on different data (files) on each iteration, so it makes sense that it would be a function.

--- rod.

Quote:

Originally Posted by tessar (Post 4802534)

This is incredible: I'm not able to copy more than one file ! :cry:

I forgot where is was in this thread, but it was suggested that you develop your own protocol (ie. packet) for transmitting the file data from A to B.

For example:

Code:

enum MsgType { NEW_FILE, DATA, END_OF_FILE, END_OF_DIR };





struct FilePacketInfo

{

    MsgType      pktType;

    char          filename[1024];

    unsigned char data[1024];

    long          dataLen;

};

When you open/read a new file, send the structure with the appropriate MsgType indicating such. If it is just data that is in the "middle" of the file, mark is such. Use the EOF to mark the end of the file.

Something like:

Code:

struct FilePacketInfo info;

FILE* fp = fopen(filename, "rb");



if (fp)

{

    memset(info, 0, sizeof(info));



    int bytesRead = 0;



    info.pktType = NEW_FILE;

    strncpy(info.filename, filename, sizeof(info.filename));



    /* attempt to read data */

    while ((bytesRead = fread(info.data, 1, sizeof(info.data), fp)) > 0)

    {

        /* got data; record how much */

        info.dataLen = bytesRead;



        /* send the data */

        send(sock_desc, (const void*) &info, sizeof(info), 0);

        

        /* set pktType in anticipation of reading more data */

        info.pktType = DATA;

    }



    /* reached end of file; send appropriate packet to denote this state. */

    info.pktType = END_OF_FILE;

    info.dataLen = 0;



    send(sock_desc, (const void*) &info, sizeof(info), 0);



    fclose(fp);

}

...

/* repeat above for each file in the directory; consider placing the code above */

/* into a function that accepts a const char* as the filename to open/process.  */

...

/* send packet to denote end of directory */

As for the receiving end, it needs to await for the reception of FilePacketInfo objects. Once it receives one, it should check the pktType to determine what to do. For the type NEW_FILE, it needs to open a file; for DATA, just write to the file. Finally, when END_OF_FILE is reached, close the file.

Note that writing a structure in its entirety (as shown above in my example) is not ideal. Nor can you expect the code above to work for systems that employ different architectures (ie. Intel vs. PPC).

P.S. One thing I forgot to mention is that you should always treat the data you read as unsigned bytes, not signed. You never know when you may have to process a binary file.

When, on client-side, I check for "end of file" it gets stuck in a loop.

Anyway, I'm getting confused. The server writes three times on the socket (right ?)

I really dunno why but this is driving me insane.

Quote:

Originally Posted by tessar (Post 4803448)

When, on client-side, I check for "end of file" it gets stuck in a loop.

Show us the code that does the file reading and checking for EOF. I thought you said you had this much working already.
--- rod.