why does SCHED_FIFO task on one CPU affects other CPU tasks on a multy core system?
 

I have a 4core maching(hyperthreading disabled), SUSE linux 2.6.32, SMP & RT group scheduling enabled, at the boot time all the OS/kernel and other applications are limited to first 2 cores. (using isolcpu)
my need is that I have 2 CPU bound processes (p1 & P2), I want them to 100% occupy the next 2 CPUs separately. And to give them SHED_FIFO policy with 99 priority.

I have successfully bound the processes with sched_setaffinity. Initially P1 & P2 are both working almost 100% CPU (with SCHED_OTHER) each on CPU3 & CPU4 respectively.

But the problem comes when they are given SCHED_FIFO. When P1 is set to SHED_FIFO (still p2 on SCHED_OTHER),
chrt -f -p 99 (pid of P1)
The whole system become almost not responding, where i cant even execute the same command for P2.

But in theory, since there are 4 CPUs, one RT task with max priority on one CPU should not affect the tasks or the OS on other CPUs.

Im trying to understand why this happens, any one have an idea? ( why just one SHED_FIFO task freezes a whole multy core system?)

I'm really stuck with this problem, expecting some expert advice.
Thankx in advance.

Why do you need SCHED_FIFO for them? There's some kernel stuff that needs to be running from time to time (workqueues...) and if your threads are not releasing the CPUs, you may get the result as observed. Normally, a SCHED_FIFO thread will run just for a small amount of time and release the CPU.

The need is, I want to give p1 & p2 maximum CPU time and minimum latency.
But do they need to run on each CPU ?, what I was thinking it that, since I have delegated two CPUs for OS and kernel, Other CPUs will not be needed by the OS/kernel. Am I wrong ?

Thankx.