Process bound to specific CPU. Are the related TCP syscalls executed on same CPU?

perec-jar · 01-04-2015, 04:05 PM

Hello, dear Community,

First of all, thanks that you are there and helping such newbies as me in answering linux questions.
And second of all, sorry for relatively long post

Before I will ask my question, I would like to describe my setup:

I have a client and server applications running on different host-mashines connected over an ethernet cable and communicating over a TCP protocol.

1. The client sends a packet to the server
2. server responds with the same packet after some delay.
3. The client measures time passed between sending own packet and receiving a packet from the server in ticks (later converted to nanoseconds).

The time measurements must be as precise as possible. I use the following article to do measurements.

I see two other aspects that must be observed to achieve the best precision:

1) The client process must be active all the time during the communication, i.e. not interrupted by other processes.
This can be achieved with the FIFO scheduling policy for this process, and it has to be assured, that:

This process does not access any synchronized resources

It has highest priority over all other processes

2) Related system calls are executed on the same core to avoid that CPU executes some other syscalls not related to this process.

So here are my questions:

1 If affinity of a specific CPU is set to execute my process, are the related syscalls (creating socket, sending packet, receiving packet) also executed on that CPU?
2 If not is it possible to achive that?
3 If not, I know that there is a PREEMPT-RT patch for linux kernel, that allows preempt CPU even when it executes a system call. Is it then a better solution?
4 I read somewhere, that setting a highest priority to my process can be a bad Idea. What can go wrong, when I do that, or what would be a better solution?

I would like to thank you beforehand for your help!

Here is a code snippet, that does the above (if I don't miss something).

Code:

void enable_real_time(){

	id_t pid = getpid();

	// set a process to use one of the real-time schedulers
	int policy = SCHED_FIFO;	// type of the real-time scheduler. this one preempts an executing process if it priority is lower that the priority of this (out) process
	struct sched_param param;
	param.__sched_priority = sched_get_priority_max(policy);	// set highest priority to make sure our process is started without a delay
	if(sched_setscheduler(pid, policy, &param) != 0 ){
		fprintf(stderr,"\nError [enable_real_time]: failed setting real-time scheduler\n");
		perror("sched_setscheduler");
	}
	else fprintf(stderr,"\nSuccess [enable_real_time]: Real-time scheduler is set\n");

	fprintf(stderr,"PID: %d\n",pid);
	fprintf(stderr,"Priority of this process: %d\n",param.__sched_priority);
}

Code:

void set_cpu_affinity(int cpu){

	unsigned long int mask = 0;
	mask = 1<<cpu;
	unsigned int len = sizeof(mask);

	if(sched_setaffinity(0, len, (cpu_set_t*) &mask) != 0){
		fprintf(stderr,"\nError [set_cpu_affinity]: failed setting affinity\n");
		perror("sched_setaffinity");
	} else {
		fprintf(stderr,"\nSuccess [set_cpu_affinity]: Processor affinity is set\n");
		fprintf(stderr,"CPU used by this process: %d\n", sched_getcpu());
	}
}

smallpond · 01-05-2015, 09:22 AM

Answers to your questions:

1. Everything that runs under the process will observe the CPU affinity, including syscalls.

Affinity tells the scheduler to only run that process on the selected CPU, however it does not prevent the scheduler from running other processes on the same CPU. To do that, you can use taskset to set their masks to be the inverse (all except that CPU).

Interrupts, such as the code that sends and receives packets, have their own method of choosing a CPU using the smp_affinity proc file. See:
http://www.thegeekstuff.com/2014/01/linux-interrupts/

2. n/a

3. The RT patch is only needed when you have more realtime processes that need low latency than you have free cores. It guarantees that you can preempt other long running processes to run your realtime processes. If you can dedicate a core to each realtime process, you don't need the RT patch.

4. The bad thing that can happen is that other processes get starved. If you dedicate a CPU to your process, then you don't need to worry about its priority. Feel free to make its niceness -20.

perec-jar · 01-05-2015, 01:19 PM

Hello, smallpond,

I am very grateful for your reply and time you dedicated to reading and responding to my questions. Your answers are indeed very useful!

As to your answer:

1)

Quote:

Affinity tells the scheduler to only run that process on the selected CPU, however it does not prevent the scheduler from running other processes on the same CPU. To do that, you can use taskset to set their masks to be the inverse (all except that CPU).

I was aware of that problem, that other processes can actually be executed on the dedicated core. Your solution is very elegant! I have done it differently perhaps even wrong (would be glad if you comment on that).
Having 4 cores on my CPU I have done the following:

Code:

1 $ cd /etc/default/
2 $ sudo gedit grub 
3 I eddited the GRUB_CMDLINE_LINUX_DEFAULT option in /etc/default/grub by appending isolcpus="3"
4 $ sudo update-grub; sudo reboot

The above prevents kernel from scheduling other processes on that core. However, my process can be scheduled explicitly on isolated core.

After reboot I checked with

Code:

for i in $(pgrep -f ".*"); do ps -mo pid,tid,fname,user,psr -p $i;done

that most of the processes were running on first 3 cores and my process (using the code from the previous post) as well as few other were running on the 4-th core (don't know why).

3)

Quote:

If you can dedicate a core to each realtime process, you don't need the RT patch.

This is particularly related to me because in another scenario I run the same code on my raspberry pi B+ that has raspbian OS on it. This chip has one CPU and my single realtime process have to share the CPU with other processes. So RT patch is probably what I need!

4)

Quote:

The bad thing that can happen is that other processes get starved.

If I run my code on raspberry pi that has one core and RT patched kernel, then does it make sense to care about the priority and preemption? If I put to sleep my realtime process between measurements, can this solve the problem with other processes' starvation?

Thanks again for your help, it is really helpful!