Hi All,

I have a server which is a fairly busy gateway. It's primary job is routing network packets based on hundreds of specially tuned rules to ensure a packet spends as little time as possible in the processing stack before being either forwarded or dropped.

Recently I've got the feeling that the system's beginning to get to its upper limits and I'm unable to pinpoint exactly what may be the bottleneck.

The system's running at an average CPU load of between 56% (load avg 4.5 on 8-cores) and 65% (load avg 5.2 on 8-cores). The memory usage of 32GB total system memory shows roughly 11GB free and the IO stats give a 'tps' value of 37.

My thinking is that the CPU's are starting to get on the verge of being too loaded and would likely be the candidate for being upgraded. The other stats seem within reason as far as I can gauge.

Am I on the right track with this or is there something else I should perhaps be looking at?

Thanks
Anubis.

I think you're on the right track.

I'd consider either a specific network appliance to perform these routing functions, unless they are so customized that a standard product cannot do this for you. Or I'd consider not allowing that server to be used for other tasks besides this primary one of performing the routing.

All that I'd guess matters depending whether or not the server is just being overloaded due to higher and higher traffic volumes versus for other reasons.

Hi Rtmistler,

Thanks for the reply, that coincides with my thinking.

The application is very specific, it's my own programming which I've already spent considerable time tuning for maximum performance system and code wise, I was just more confirming my suspicions as to the next bottleneck component to gauge the upgrade level.

The server its self only does this one job, no other sundry tasks, so it looks like either a CPU upgrade or system upgrade may be in order.

Thanks
Anubis.

There comes a point where customized hardware assist is the best answer. For instance when I worked on a high speed GigE switch, the chips were specialized hardware which allowed us to rapidly examine and edit the MAC and IP headers so we could then give directions to the chip as to what to do with the packet. Meanwhile the packet contents were only ever moved twice by the hardware, once in, once out.

A problem you're facing is that you have general hardware, a hardware interface driver, Linux, and so forth and those packets are being moved probably 6 or more times prior to being finally decided upon. The custom designs have DMA controlled by intelligent comm chips like I'm describing above, hence they can operate with minimal copy overhead and it's a great performance benefit.

If you can, take in a packet's contents. Design a structure where you can look at the headers and modify them, but not move the payload around, you'll have to determine how to deal with varying packet header sizes, meaning you'll have to allocate larger than the packet when you ingress it. Don't copy it around as you pass it through your decision flow, and try to make the only copies be ingress from the hardware driver, and egress back out.

Allocate at startup your worst case buffer amount and manage the buffers instead of allocating and freeing all the time, if you happen to be doing something like that. Plus you'll be more able to detect when you reach or come close to system limitations if you monitor your use of that buffer pool.

Design in "life of a packet" time metrics measurements in your architecture. Again, when you determine that the life of a packet timeframe is growing, you need to determine where the bottleneck is, and if that's software.

Determine the amount of time the CPU is used by your program.

If you're writing the software, there's a lot you can do. But yes eventually you will hit limitations for memory and CPU capabilities.

Hi Rtmistler,

Thanks again for the detailed reply, that's certainly some very good food for thought and is a direction I am thinking of heading in future-wise; though I would need to source somewhere to design such a specific hardware appliance.

There are still some things I can do software wise to mitigate some additional load, but this is at the expense of an additional 'box' to move that load onto (separating tasks) as I've already trimmed things down to there smallest processes; however, I would like to avoid that if possible, it's a much simpler idea to keep it all in one box.

If you have any good sources where I could discuss such a hardware appliance, I would certainly like to know.

Thanks yet again for your valuable input.
Anubis.

Can't really offer much in the way of open source. Those experiences were for like Cisco and Nortel where they actually made the chips or partnered with a semiconductor manufacturer to get the customized chips.

I'm sure there are chips out there for purchase, but you'd have to then create a board, because some reference design is only going to get you so far. Meanwhile they may not have a viable reference design anyways, you really would have to design the board.