Folks,

I have written a multiprocess application that uses message queue to communicate with each other.

Here is what I am doing, when I start my application it creates a message queue and forks a process. The child process reads multicast packets from the network and writes to message queue. The parent process reads packets from message queue and compares source ip and sequence number (it is part of payload) with last 64K packets received to see if it has received a duplicate packet. I am using message queue as a buffer because I do not want child process to drop any packets while it is comparing it with previously received packets. The message queue is large enough to contain 64K packets. To compare the old packets I am using array of structures as circular buffer. During a spike I may receive 100 - 120 packets per milli second.

When I run my application, the parent process keeps up with the child process, I can see that with "ipcs -q". After about 30 seconds it cannot keep up and the size of message queue keeps increasing until it is full. When I run "top" I can see that one CPU/core is hundred percent busy while other 7 cores are idle. It seems that both processes are running on same core and the child process gets interrupts everytime there is a packet on the net and starves the parent process.

I am running RHEL 5. The system has 24GB memory and my application is the only application running on it. It is a HP G6 server.

How can I find out if they are both running on same CPU/core?
How can I force them to run on different CPU?

I tried to use taskset command but it did not make any difference.

Thanks.

Do you have irqbalance running ? .. this will distribute interrupts across all cpu's

cheers

I will check when I get back to office.

Thanks.

Regardless of the interrupts (independent) userspace processes should be farmed out across engines.
I was thinking maybe a coding problem - multi-threaded isn't automatically multi-processor aware (for example, how are you managing the queue ?).
But taskset not helping is a worry - did you check the return code to make sure it actually worked ?. I don't use taskset, I use cgroups to separate work, but children would be an issue (I usually isolate them to the same CPU subset). Could be done by pid.
"top" will show you the CPU last dispatched on - once running, enter "f" (lower case) then "j". Can be saved to a config file - see the manpage.
Another (better) option would be to look at installing collectl - it has very good processes based data if you have the kernel support in.