Hi. Hard question here. I'm programming a software system that consists of multiple processes. It is programmed in C++ under Linux. and they communicate among them using Linux shared memory.

Usually, in software development, is in the final stage when the performance optimization is made. Here I came to a big problem. The software has high performance requirements, but in machines with 4 or 8 CPU cores (usually with more than one CPU), it was only able to use 3 cores, thus wasting 25% of the CPU power in the first ones, and more than 60% in the second ones.
After many, many research, and having discarded mutex and lock contention, I found out that the time was being wasted on shmdt/shmat calls (detach and attach to shared memory segments). After some more research, I found out that these CPUs, which usually are AMD Opteron and Intel Xeon, use a memory system called NUMA, which basically means that each processor has its fast, "local memory", and accessing memory from other CPUs is expensive.

After doing some tests, the problem seems to be that the software is designed so that, basically, any process can pass shared memory segments to any other process, and to any thread in them. This seems to kill performance, as process are constantly accessing memory from other processes.

Now, the question is, is there any way to force groups of process to execute in the same CPU?. I don't mean to force them to execute always in the same processor, as I don't care in which one they are executed, but that would do the job. Ideally, there would be a way to tell the kernel: If you schedule this process in one processor, you must also schedule this "brother" process (which is the process with which it communicates through shared memory) in that same processor, so that performance is not penalized.


Hard one, I know, as probably very few people know that well the Linux scheduling system. But after countless days working into this, anyone suggesting anything that could remotely help would be greatly appreciated.

There are a few issues here. If you are (truly) running separate processes, then there is unlikely to be any ability for the scheduler to localise to the same NUMA node. If you are using threads, they should be dispatched on the same node as the parent - generally.
Regardless, you can use cgroups to enforce dispatch where-ever you want. It uses groups of processors, and can be setup to "catch" forked processes as well. Perfect fit for what you want. I use it to separate benchmark and monitoring tests.
Documentation used to be (only) in the kernel source tree - try a google for cgroups and see what turns up.

Edit: forgot to mention, you can enforce the memory locality to particular nodes too.

1 members found this post helpful.