Latest LQ Deal: Complete CCNA, CCNP & Red Hat Certification Training Bundle
Go Back > Forums > Non-*NIX Forums > Programming
User Name
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.


  Search this Thread
Old 11-26-2011, 06:32 PM   #1
LQ Newbie
Registered: Jan 2010
Location: France, Europe
Distribution: Debian
Posts: 2

Rep: Reputation: 0
vectored write to many sockets (with tee()/splice() ?)

Hello !

In the process of writing a lightweight media server, I'm investigating ways or reducing context switching and memory copies with Linux.

Here is the kind of code that is consuming a lot of CPU. Basically, it's multicasting data from one TCP socket to many others (thousands).
recv(src_fd, buf, len);
mangle_my_data(buf, len);
for (i=0; i<dst_count; i++) {
  send(dst[i], buf, len);
One important point being the buffer manipulation before the data gets written, and as far as I understand splice() it keeps me from using that call because the source must be a file descriptor or pipe.

So it could take advantage of zero-copy syscalls but I can't see the best way to implement that, and I'm felling like there could be other great features less documented in splice() and tee().
Unfortunatelly, vectored AIO do not work on sockets ...

Currently, my best option was to replace the buffer by a pipe, and let tee() feed the clients. It's still making many syscalls that I hope can be vectored by some mean, but it's saving memory !
recv(src_fd, buf, len);
mangle_my_data(buf, len);
write(pipe_fd, buf, len);
for (i=0; i<dst_count; i++) {
  tee(pipe_fd, dst[i], len);
// and ultimately pop data from the pipe

So do you guys know the best way to write the same data to 10k sockets using zerocopy and vectored I/O ?


Last edited by tibal; 11-26-2011 at 07:00 PM. Reason: fixed code example
Old 11-26-2011, 11:13 PM   #2
Nominal Animal
Senior Member
Registered: Dec 2010
Location: Finland
Distribution: Xubuntu, CentOS, LFS
Posts: 1,723
Blog Entries: 3

Rep: Reputation: 947Reputation: 947Reputation: 947Reputation: 947Reputation: 947Reputation: 947Reputation: 947Reputation: 947
If you are sending the exact same data to multiple clients, you should consider using multicast over TCP/IP.

If you are using Linux 2.6.31 or later (including 3.0+), you can use the PACKET_MMAP interface to send a large number of packets using a single syscall. (The patch mentioned at the wiki page are included in current Linux kernels.) You do need to generate the IP headers yourself for each packet, but you can find examples and links at the wiki page.

Since syscall overhead is a major factor, especially so with small packets, using PACKET_MMAP should greatly reduce CPU use, and allow you to reach full bandwidth usage if necessary.


linux, optimization, programming, server, socket

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off

Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] Splice syscall does not work vadiml Programming 7 10-15-2015 07:43 AM
Parial Write for sockets in LINUX sanushchacko Linux - Networking 5 05-19-2011 04:05 AM
How to write a log file of a user interactive process using tee. ira Linux - Newbie 2 10-01-2010 03:55 AM
Program to forward tcp sockets to unix domain sockets mikepol Linux - Networking 0 09-27-2007 09:49 AM
Using write fn for sockets in both BSD and Linux trotters78 Programming 0 10-15-2004 03:37 AM > Forums > Non-*NIX Forums > Programming

All times are GMT -5. The time now is 04:14 PM.

Main Menu
Write for LQ is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration