LinuxQuestions.org
Share your knowledge at the LQ Wiki.
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices

Reply
 
LinkBack Search this Thread
Old 05-02-2012, 09:56 AM   #1
bayoulinux
Member
 
Registered: Oct 2011
Posts: 34

Rep: Reputation: Disabled
Linux packet sockets basic question - bi-directional raw ethernet data flow


Hello:

Newbie Linux socket programming question... I've successfully created a short user space program that passes raw ethernet packets around via the socket()/bind() APIs:

socket(AF_PACKET, SOCK_RAW, htons(ETH_P_ALL));
bind()

Basically all the program does is to pass along from one Ethernet interface/socket() and out another Ethernet interface/socket(). For lack of a better picture:

----> packets in ----> Eth_if_in ---> sock_in ---> sock_out ---> Eth_if_out --->

So just unidirectional packet passing... Now my newbie question is if I want to edit my 20 line C program to pass traffic in a bi-directional manner across the same interfaces, do I need to create an additional two sockets, or .... what's the smartest way to get the raw ethernet traffic passed along the same two Ethernet interfaces?

Thank you
 
Old 05-03-2012, 08:11 PM   #2
dwhitney67
Senior Member
 
Registered: Jun 2006
Location: Maryland
Distribution: Kubuntu, Fedora, RHEL
Posts: 1,494

Rep: Reputation: 327Reputation: 327Reputation: 327Reputation: 327
You can send and receive using one socket.
 
Old 05-04-2012, 04:40 AM   #3
slzckboy
Member
 
Registered: May 2005
Location: uk - Reading
Distribution: slack 10.2 kde 3.4.2 kernel 2.6.15
Posts: 452

Rep: Reputation: 30
tcp by definition is bi directional.
 
Old 05-04-2012, 03:17 PM   #4
theNbomr
LQ 5k Club
 
Registered: Aug 2005
Distribution: OpenSuse, Fedora, Redhat, Debian
Posts: 5,388
Blog Entries: 2

Rep: Reputation: 900Reputation: 900Reputation: 900Reputation: 900Reputation: 900Reputation: 900Reputation: 900Reputation: 900
Quote:
Originally Posted by slzckboy View Post
tcp by definition is bi directional.
Except these are not TCP or UDP sockets. They are, however bidirectional, as a previous poster points out.
You can read & write datagrams on each of the sockets. You will almost certainly need to use something like select() to determine which socket is the first to have incoming data.
--- rod.
 
Old 05-05-2012, 03:33 AM   #5
Nominal Animal
Senior Member
 
Registered: Dec 2010
Location: Finland
Distribution: Xubuntu, CentOS, LFS
Posts: 1,723
Blog Entries: 3

Rep: Reputation: 942Reputation: 942Reputation: 942Reputation: 942Reputation: 942Reputation: 942Reputation: 942Reputation: 942
Unless you use two separate buffers, you may find that your ethernet bridge deadlocks: all three (both endpoints and your bridge) may be waiting for more data to arrive at the same time.

The solution is to use a write buffer for each socket, and always prefer writes over reads when there is stuff in the buffer for that socket. Data read from a socket is simply appended to the other buffer. To avoid any nasty corner cases, I'd recommend setting both sockets nonblocking (using fcntl(,F_GETFL,) followed by fcntl(,F_SETFL,)), and always trying a nonblocking write()/send()/sendmsg() on both sockets if there is data in the respective buffer, before doing a select() to see which socket can be read from or written to.

If you want much simpler code, you can instead use two threads, each reading from one socket and writing the read data to the other socket, using normal, blocking I/O.
 
Old 05-06-2012, 08:29 AM   #6
bayoulinux
Member
 
Registered: Oct 2011
Posts: 34

Original Poster
Rep: Reputation: Disabled
Hello:

Thank you for the replies. I'm interested in:

"If you want much simpler code, you can instead use two threads, each reading from one socket and writing the read data to the other socket, using normal, blocking I/O."

Do you mind posting some pseudo code that I could use to implement this model?

Thank you
 
Old 05-06-2012, 11:45 AM   #7
Nominal Animal
Senior Member
 
Registered: Dec 2010
Location: Finland
Distribution: Xubuntu, CentOS, LFS
Posts: 1,723
Blog Entries: 3

Rep: Reputation: 942Reputation: 942Reputation: 942Reputation: 942Reputation: 942Reputation: 942Reputation: 942Reputation: 942
Here is a very simple threaded code skeleton, with the associated "work descriptor":
Code:
#define  _POSIX_C_SOURCE 200809L
#define  _GNU_SOURCE
#include <unistd.h>
#include <stdlib.h>
#include <pthread.h>
#include <errno.h>

/* C99: work descriptor and data buffer.
*/
struct work {
    int     source;   /* Descriptor to read from */
    int     target;   /* Descriptor to write to */
    size_t  size;     /* Size of data array, in chars */
    char    data[];   /* C99: Dynamically sized array. */
};

/* Worker thread function.
 * Payload is a pointer to a work descriptor.
*/
void *worker(void *const payload)
{
    struct work *const work = (struct work *)payload;
    ssize_t            bytes;

    while (1) {

        do {
            bytes = read(work->source, work->data, work->size);
        } while (bytes == (ssize_t)-1 && errno == EINTR);

        /* End of input? */
        if (bytes == (ssize_t)0)
            return (void *)0L;

        /* Read error? */
        if (bytes == (ssize_t)-1)
            return (void *)((long)errno);

        /* C library I/O error? */
        if (bytes < (ssize_t)1)
            return (void *)((long)EIO);

        /* Write bytes data to target. */
        {
            const char       *head = work->data;
            const char *const tail = work->data + bytes;

            while (head < tail) {

                do {
                    bytes = write(work->target, head, (size_t)(tail - head));
                } while (bytes == -1 && errno == EINTR);

                /* Write error? */
                if (bytes == (ssize_t)-1)
                    return (void *)((long)errno);

                /* Library I/O error? */
                if (bytes < (ssize_t)1)
                    return (void *)((long)EIO);

                /* Successful (partial) write. */
                head += bytes;
            }
        }
    }
}
First, you need to create the two work descriptors. For example, allowing for any size Ethernet frames (including jumbo frames -- total size about 9042 bytes or so; round to 0x2360 = 9056 bytes total):
Code:
    const size_t size = 9056;
    struct work *work1, *work2;

    work1 = malloc(sizeof (struct work) + size);
    work2 = malloc(sizeof (struct work) + size);
    if (!work1 || !work2) {
        /* Out of memory, abort! */
        exit(1);
    }

    work1->source = socket1_descriptor;
    work1->target = socket2_descriptor;
    work1->size   = size;

    work2->source = socket2_descriptor;
    work2->target = socket1_descriptor;
    work2->size   = size;
Next, you create the worker threads:
Code:
    pthread_t       worker1, worker2;
    pthread_attr_t  attr;
    int             result;

    /* Clear thread attribute set. */
    result = pthread_attr_init(&attr);
    if (result) {
        /* Cannot initialize thread attributes; errno in result. Abort. */
        exit(1);
    }

    /* The default thread stack size is way too large,
     * it just wastes a lot of memory. We can do with
     * a very, very small stack. Use 16384 bytes. */
    pthread_attr_setstacksize(&attr, (size_t)16384);

    result = pthread_create(&worker1, &attr, worker, work1);
    if (result) {
        /* Cannot create a thread; errno in result. Abort. */
        exit(1);
    }

    result = pthread_create(&worker2, &attr, worker, work2);
    if (result) {
        /* Cannot create a thread; errno in result. Abort. */
        exit(1);
    }

    /* The thread attribute object is no longer needed. */
    pthread_attr_destroy(&attr);
Note that they will start working immediately, and will work forever, or until the other ends close the sockets. Setting the stack size like I do above is not really necessary, but it is very good practice: the default thread stack size is very large (2 MB, typically). Still, remember that only the stack for the main thread grows automatically; any thread you create will only have the fixed size stack. The above worker threads need minimum stack, so it is a good idea to minimize their stack size.

The main thread is free to do whatever it wants. While the main thread could simply pthread_join() the threads, you can only block on one thread, and you won't notice if the other exits first. Therefore, to wait for the threads to exit, I recommend using
Code:
    struct timespec interval;
    int             err1 = EBUSY;
    int             err2 = EBUSY;
    void           *resultptr;

    /* Sleep interval between checks.
     * Note: 1000000000 nsec = 1 second.
    */
    interval.tv_sec = 1;
    interval.tv_nsec = 0L;

    while (err1 == EBUSY || err2 == EBUSY) {

       /* Check if worker 1 has exited. */
       if (err1 == EBUSY) {
           result = pthread_tryjoin_np(worker1, &resultptr);
           if (!result) {
               err1 = (long)resultptr;

               /* You can report worker 1 has exited;
                * just remember to fflush(stdout)/fflush(stderr)
                * to make sure it gets output. */


               continue;
           }
       }

       /* Check if worker 2 has exited. */
       if (err2 == EBUSY) {
           result = pthread_tryjoin_np(worker2, &resultptr);
           if (!result) {
               err2 = (long)resultptr;

               /* You can report worker 2 has exited;
                * just remember to fflush(stdout)/fflush(stderr)
                * to make sure it gets output.
               */

               continue;
           }
       }

       /* Nothing interesting is happening.
        * Sleep for a bit, then recheck.
       */
       nanosleep(&interval, NULL);
    }
Note that if/when a worker threads exits, you notice it (within the interval), but the code will only exit the loop when both threads have exited.

I used nanosleep() because it does not interfere with signals (especially the ALRM signal). In Linux, you could just as well use the usleep() function, because it uses the same kernel facility to sleep. There are other ways to sleep for a while, too; it does not matter which you use, as long as you note the side effects. (I used nanosleep() above simply because it has NO side effects at all.)

If you were to run the code on an embedded system, or for other reasons wish to eliminate the sleep, you could use a semaphore. When a thread exits, it would post the semaphore. The main thread would wait on the semaphore, and thus block until one of the threads exits. This way your code would really not use any extra CPU time.

(However, I wanted to keep things simple, and leave easy things for you to implement and enhance the code, as an incentive.)

You should be able to use the above code to stitch a working solution together. There really is not that much code missing, which is why I wrote it in pieces. I really want you to understand how they work, learn for yourself, and adapt it to your own use. Copying a ready solution is rather dull.

If you have any specific questions or issues (parts of the code are untested!), I'll be happy to elaborate.

Hope you find this useful,

Last edited by Nominal Animal; 05-06-2012 at 11:51 AM.
 
Old 05-07-2012, 08:17 AM   #8
bayoulinux
Member
 
Registered: Oct 2011
Posts: 34

Original Poster
Rep: Reputation: Disabled
Wow - thank you very much for posting such a detailed response. I really appreciate it.

I was about to comment about instead of "blindly" sleeping, I'd want to sleep and awake based upon an event (as to not waste CPU time). However, I see you have covered that as well within your description.

Let me try this out and do some self study... Thank you again for taking the time to write up so many details!
 
Old 05-07-2012, 03:54 PM   #9
bayoulinux
Member
 
Registered: Oct 2011
Posts: 34

Original Poster
Rep: Reputation: Disabled
Got it all working today - thanks again!
 
Old 05-14-2012, 04:01 PM   #10
bayoulinux
Member
 
Registered: Oct 2011
Posts: 34

Original Poster
Rep: Reputation: Disabled
Hi:

I had a follow on question regarding Nominal Animal's generous code posting...

With this model is there a concern regarding thread safety and the use of the same file descriptors in two separate threads?

work1->source = socket1_descriptor;
work1->target = socket2_descriptor;
work1->size = size;

work2->source = socket2_descriptor;
work2->target = socket1_descriptor;
work2->size = size;

Anything regarding this file descriptor and socket assignment used in two different threads.... I'm actually using sendto() and recvfrom() if that matters...?

Thank you!
 
Old 05-14-2012, 05:18 PM   #11
Nominal Animal
Senior Member
 
Registered: Dec 2010
Location: Finland
Distribution: Xubuntu, CentOS, LFS
Posts: 1,723
Blog Entries: 3

Rep: Reputation: 942Reputation: 942Reputation: 942Reputation: 942Reputation: 942Reputation: 942Reputation: 942Reputation: 942
(If anyone disagrees with the following, please do pipe up! I do make errors, although I try hard not to. If I am in error, I'd appreciate being corrected.)

Quote:
Originally Posted by bayoulinux View Post
With this model is there a concern regarding thread safety and the use of the same file descriptors in two separate threads?
With regards to file descriptors in general, there is, yes. Good catch!

Things like file position, file or range locks, leases, and so on, must be carefully managed when the same descriptor is used in multiple threads.

Considering side effects, a socket pair (meaning two connected sockets) is very similar to a pair of pipes, due to the inherent bidiectionality of sockets. Not just in functionality, but in how independent the "sides"/"directions" are.

In this particular case, the descriptors are sockets, and therefore inherently bidirectional. Furthermore, each socket is only read from a single thread, and written to by a single thread.

If you had phrased your question as "the use of the same socket descriptors in two separate threads?", I could have simply answered that no, there is no issue. It is safe to handle writing to a connected socket from one thread, and reading from it from another. Indeed, it is a relatively common technique, well suited to certain approaches to server implementations.

The read and write functionalities of a connected socket are almost completely independent. If I recall correctly, they share the descriptor number, the kernel buffer, the socket options, and the connected endpoint. Nothing that would interfere in a situation like this. You can even close one side of the socket independently from the other, using[FONT=Monospace] shutdown()/FONT].

If you used the same exact code with e.g. two files (two ordinary file descriptors), you will get unpredictable results. (Specifically, the results will depend on thread timing issues, since each file descriptor only has one position, and both reading from it and writing to it will advance the file position. To be honest, I don't really even want to think about exactly what would happen.. the files would probably grow sparsely until they consumed all disk space.)

With FIFOs or named pipes, the code should work fine, although normal FIFO caveats apply. (In particular, opening a fifo will block until another process or threads opens the complementary side. Multiple readers must cooperate to not read at the same time, or a random one will get the next chunk.)

With character devices and other special file-like descriptors, it depends on how the read and write functionalities interact. If they are separate, the code will work as-is. If they share properties, like some sort of a logical position for example, then the code is not safe to use for that.

In summary: It is not a safe technique for file descriptors in general due to the many properties the read and write sides share. For sockets, it is a safe, commonly known technique, since the read and write sides are practically independent.

I'd also like to say it warms my heart to see somebody actually bothering to ponder such issues. If you have any further questions, I for one will be happy to try to help.
 
Old 05-16-2012, 11:54 AM   #12
bayoulinux
Member
 
Registered: Oct 2011
Posts: 34

Original Poster
Rep: Reputation: Disabled
Hi:

Thanks again for the responses!

So, I think I understand... but just to review:

work1->source = socket1_descriptor;
work1->target = socket2_descriptor;

work2->source = socket2_descriptor;
work2->target = socket1_descriptor;

There are only two socket descriptors created. Both of these socket descriptors are used within two different threads. So there isn't any mutex protection needed when one thread is using socket1_descriptor to read from, and another thread is using the same socket1_descriptor to write to?

Is that right?

Thanks!
 
Old 05-16-2012, 03:24 PM   #13
Nominal Animal
Senior Member
 
Registered: Dec 2010
Location: Finland
Distribution: Xubuntu, CentOS, LFS
Posts: 1,723
Blog Entries: 3

Rep: Reputation: 942Reputation: 942Reputation: 942Reputation: 942Reputation: 942Reputation: 942Reputation: 942Reputation: 942
Quote:
Originally Posted by bayoulinux View Post
There are only two socket descriptors created. Both of these socket descriptors are used within two different threads. So there isn't any mutex protection needed when one thread is using socket1_descriptor to read from, and another thread is using the same socket1_descriptor to write to?
Correct.

No protection is needed, because writing to a socket does not interfere with reading from that same socket in any way. It is safe for one thread to read from the socket, and another thread to write to that same socket; no protection is needed.

Last edited by Nominal Animal; 05-16-2012 at 03:26 PM.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off
Trackbacks are Off
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Raw sockets for layer 2 ethernet on a 2.4 kernel rjcarlson49 Linux - Kernel 1 06-20-2010 12:14 AM
Receive Ethernet VLAN frame on the native ethernet interface (raw packet socket) scottbiker Linux - Networking 0 07-13-2009 03:49 AM
Reading the ip address from a packet raw sockets linuxtest Programming 2 12-10-2008 02:43 PM
Raw Packet Sockets in Fedora Core 6 don't transmit Son_of_Merlin Linux - Networking 6 01-02-2007 12:55 AM
Raw Ethernet Sockets alanwolfen Programming 2 01-06-2005 06:51 PM


All times are GMT -5. The time now is 11:13 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration