LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices

Reply
 
Search this Thread
Old 11-26-2011, 06:32 PM   #1
tibal
LQ Newbie
 
Registered: Jan 2010
Location: France, Europe
Distribution: Debian
Posts: 2

Rep: Reputation: 0
vectored write to many sockets (with tee()/splice() ?)


Hello !

In the process of writing a lightweight media server, I'm investigating ways or reducing context switching and memory copies with Linux.

Here is the kind of code that is consuming a lot of CPU. Basically, it's multicasting data from one TCP socket to many others (thousands).
Code:
recv(src_fd, buf, len);
mangle_my_data(buf, len);
for (i=0; i<dst_count; i++) {
  send(dst[i], buf, len);
}
One important point being the buffer manipulation before the data gets written, and as far as I understand splice() it keeps me from using that call because the source must be a file descriptor or pipe.

So it could take advantage of zero-copy syscalls but I can't see the best way to implement that, and I'm felling like there could be other great features less documented in splice() and tee().
Unfortunatelly, vectored AIO do not work on sockets ...

Currently, my best option was to replace the buffer by a pipe, and let tee() feed the clients. It's still making many syscalls that I hope can be vectored by some mean, but it's saving memory !
Code:
recv(src_fd, buf, len);
mangle_my_data(buf, len);
write(pipe_fd, buf, len);
for (i=0; i<dst_count; i++) {
  tee(pipe_fd, dst[i], len);
}
// and ultimately pop data from the pipe

So do you guys know the best way to write the same data to 10k sockets using zerocopy and vectored I/O ?

Thanks

Last edited by tibal; 11-26-2011 at 07:00 PM. Reason: fixed code example
 
Old 11-26-2011, 11:13 PM   #2
Nominal Animal
Senior Member
 
Registered: Dec 2010
Location: Finland
Distribution: Xubuntu, CentOS, LFS
Posts: 1,723
Blog Entries: 3

Rep: Reputation: 942Reputation: 942Reputation: 942Reputation: 942Reputation: 942Reputation: 942Reputation: 942Reputation: 942
If you are sending the exact same data to multiple clients, you should consider using multicast over TCP/IP.

If you are using Linux 2.6.31 or later (including 3.0+), you can use the PACKET_MMAP interface to send a large number of packets using a single syscall. (The patch mentioned at the wiki page are included in current Linux kernels.) You do need to generate the IP headers yourself for each packet, but you can find examples and links at the wiki page.

Since syscall overhead is a major factor, especially so with small packets, using PACKET_MMAP should greatly reduce CPU use, and allow you to reach full bandwidth usage if necessary.
 
  


Reply

Tags
linux, optimization, programming, server, socket


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Parial Write for sockets in LINUX sanushchacko Linux - Networking 5 05-19-2011 04:05 AM
How to write a log file of a user interactive process using tee. ira Linux - Newbie 2 10-01-2010 03:55 AM
Splice syscall does not work vadiml Programming 2 10-26-2007 05:21 AM
Program to forward tcp sockets to unix domain sockets mikepol Linux - Networking 0 09-27-2007 09:49 AM
Using write fn for sockets in both BSD and Linux trotters78 Programming 0 10-15-2004 03:37 AM


All times are GMT -5. The time now is 03:35 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration