I have been building a high-throughput server application for multimedia messaging (chat/voip/control-applications). Each server can be used in stand-alone mode or many servers can be joined together to create a DHT-based overlay network. The work is in progress. Currently the server can handle around 200,000 messages per second (256 byte messages) and has a max throughput of around 256 MB/s on my machine (Intel i3 Mobile, Fedora Core 18 (64-bit), 4 GB RAM). The server has got two threads, one thread for handling all IOs (epoll-based, edge triggered) another one for processing the incoming messages. There is another Thread for Overlay Management, but doesn't matter in current discussion.
The two threads in discussion share data using two Circular Buffers. Thread no. 1 enqueues fresh messages for the Thread no. 2 using one Circular Buffer, while the Thread no. 2 returns back the processed messages through the other circular Buffer. The server is completely lock-free. I am not using any synchronization primitive what-so-ever, not even atomic operations.
The Circular Buffers never overflow because the Messages are pooled (preallocated on start), In fact all vital/frequently-used data-structures are pooled to reduce memory fragmentation and to increase cache efficiency, hence we know the maximum number of messages when the server starts, hence we can pre-calculate the maximum size of the buffers and then initialize the Circular Buffers accordingly.
Now my question: Thread #1 enqueues the serialized messages one message at a time (actually the pointers to messages), while the Thread #2 takes out messages from the queue in chunks (chunks of 32/64/128), and returns back the processed messages in chunks through the second circular buffer. In case there are no new messages Thread #2 keeps busy waiting, hence keeping one of the CPU cores busy all the time. How can I improve upon the design further? What are the alternatives for the busy wait strategy? I want to do this elegantly and efficiently. I have considered using semaphores, but I fear that is not the best solution, second good option might be signal. Yes, I did use google but none of the solutions I found on net were satisfactory. Here are a few considerations:
A. The server must spend minimum CPU cycles while making System Calls
B. There must be very low overhead
C. No Locking what-so-ever
Here is the link to the site where I have shared info about my server, to better understand the functionality and the purpose:
www.wanhive.com
This is my first post on this forum.
