LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Networking
User Name
Password
Linux - Networking This forum is for any issue related to networks or networking.
Routing, network cards, OSI, etc. Anything is fair game.

Notices


Reply
  Search this Thread
Old 01-04-2015, 04:05 PM   #1
perec-jar
LQ Newbie
 
Registered: Jan 2015
Location: Germany
Distribution: Ubuntu 12.04 LTS
Posts: 4

Rep: Reputation: Disabled
Process bound to specific CPU. Are the related TCP syscalls executed on same CPU?


Hello, dear Community,

First of all, thanks that you are there and helping such newbies as me in answering linux questions.
And second of all, sorry for relatively long post

Before I will ask my question, I would like to describe my setup:

I have a client and server applications running on different host-mashines connected over an ethernet cable and communicating over a TCP protocol.

1. The client sends a packet to the server
2. server responds with the same packet after some delay.
3. The client measures time passed between sending own packet and receiving a packet from the server in ticks (later converted to nanoseconds).

The time measurements must be as precise as possible. I use the following article to do measurements.

I see two other aspects that must be observed to achieve the best precision:

1) The client process must be active all the time during the communication, i.e. not interrupted by other processes.
This can be achieved with the FIFO scheduling policy for this process, and it has to be assured, that:
  • This process does not access any synchronized resources
  • It has highest priority over all other processes
2) Related system calls are executed on the same core to avoid that CPU executes some other syscalls not related to this process.

So here are my questions:

1 If affinity of a specific CPU is set to execute my process, are the related syscalls (creating socket, sending packet, receiving packet) also executed on that CPU?
2 If not is it possible to achive that?
3 If not, I know that there is a PREEMPT-RT patch for linux kernel, that allows preempt CPU even when it executes a system call. Is it then a better solution?
4 I read somewhere, that setting a highest priority to my process can be a bad Idea. What can go wrong, when I do that, or what would be a better solution?


I would like to thank you beforehand for your help!

Here is a code snippet, that does the above (if I don't miss something).
Code:
void enable_real_time(){

	id_t pid = getpid();

	// set a process to use one of the real-time schedulers
	int policy = SCHED_FIFO;	// type of the real-time scheduler. this one preempts an executing process if it priority is lower that the priority of this (out) process
	struct sched_param param;
	param.__sched_priority = sched_get_priority_max(policy);	// set highest priority to make sure our process is started without a delay
	if(sched_setscheduler(pid, policy, &param) != 0 ){
		fprintf(stderr,"\nError [enable_real_time]: failed setting real-time scheduler\n");
		perror("sched_setscheduler");
	}
	else fprintf(stderr,"\nSuccess [enable_real_time]: Real-time scheduler is set\n");

	fprintf(stderr,"PID: %d\n",pid);
	fprintf(stderr,"Priority of this process: %d\n",param.__sched_priority);
}
Code:
void set_cpu_affinity(int cpu){

	unsigned long int mask = 0;
	mask = 1<<cpu;
	unsigned int len = sizeof(mask);

	if(sched_setaffinity(0, len, (cpu_set_t*) &mask) != 0){
		fprintf(stderr,"\nError [set_cpu_affinity]: failed setting affinity\n");
		perror("sched_setaffinity");
	} else {
		fprintf(stderr,"\nSuccess [set_cpu_affinity]: Processor affinity is set\n");
		fprintf(stderr,"CPU used by this process: %d\n", sched_getcpu());
	}
}
 
Old 01-05-2015, 09:22 AM   #2
smallpond
Senior Member
 
Registered: Feb 2011
Location: Massachusetts, USA
Distribution: Fedora
Posts: 4,162

Rep: Reputation: 1268Reputation: 1268Reputation: 1268Reputation: 1268Reputation: 1268Reputation: 1268Reputation: 1268Reputation: 1268Reputation: 1268
Answers to your questions:

1. Everything that runs under the process will observe the CPU affinity, including syscalls.

Affinity tells the scheduler to only run that process on the selected CPU, however it does not prevent the scheduler from running other processes on the same CPU. To do that, you can use taskset to set their masks to be the inverse (all except that CPU).

Interrupts, such as the code that sends and receives packets, have their own method of choosing a CPU using the smp_affinity proc file. See:
http://www.thegeekstuff.com/2014/01/linux-interrupts/

2. n/a

3. The RT patch is only needed when you have more realtime processes that need low latency than you have free cores. It guarantees that you can preempt other long running processes to run your realtime processes. If you can dedicate a core to each realtime process, you don't need the RT patch.

4. The bad thing that can happen is that other processes get starved. If you dedicate a CPU to your process, then you don't need to worry about its priority. Feel free to make its niceness -20.
 
1 members found this post helpful.
Old 01-05-2015, 01:19 PM   #3
perec-jar
LQ Newbie
 
Registered: Jan 2015
Location: Germany
Distribution: Ubuntu 12.04 LTS
Posts: 4

Original Poster
Rep: Reputation: Disabled
Using both RT patch and preemption on a single core CPU

Hello, smallpond,

I am very grateful for your reply and time you dedicated to reading and responding to my questions. Your answers are indeed very useful!

As to your answer:

1)

Quote:
Affinity tells the scheduler to only run that process on the selected CPU, however it does not prevent the scheduler from running other processes on the same CPU. To do that, you can use taskset to set their masks to be the inverse (all except that CPU).
I was aware of that problem, that other processes can actually be executed on the dedicated core. Your solution is very elegant! I have done it differently perhaps even wrong (would be glad if you comment on that).
Having 4 cores on my CPU I have done the following:
Code:
1 $ cd /etc/default/
2 $ sudo gedit grub 
3 I eddited the GRUB_CMDLINE_LINUX_DEFAULT option in /etc/default/grub by appending isolcpus="3"
4 $ sudo update-grub; sudo reboot
The above prevents kernel from scheduling other processes on that core. However, my process can be scheduled explicitly on isolated core.

After reboot I checked with
Code:
for i in $(pgrep -f ".*"); do ps -mo pid,tid,fname,user,psr -p $i;done
that most of the processes were running on first 3 cores and my process (using the code from the previous post) as well as few other were running on the 4-th core (don't know why).

3)
Quote:
If you can dedicate a core to each realtime process, you don't need the RT patch.
This is particularly related to me because in another scenario I run the same code on my raspberry pi B+ that has raspbian OS on it. This chip has one CPU and my single realtime process have to share the CPU with other processes. So RT patch is probably what I need!

4)
Quote:
The bad thing that can happen is that other processes get starved.
If I run my code on raspberry pi that has one core and RT patched kernel, then does it make sense to care about the priority and preemption? If I put to sleep my realtime process between measurements, can this solve the problem with other processes' starvation?

Thanks again for your help, it is really helpful!
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Profiling of LInux box With respect to CPU and Network usage by specific process vlrk Linux - Server 0 10-31-2013 11:44 PM
LXer: How to run program or process on specific CPU cores on Linux LXer Syndicated Linux News 0 10-29-2013 10:30 AM
executing a cpu bound multithreaded process takzee Linux - Newbie 3 04-14-2013 08:30 PM
How can you force a program/process to run on a specific CPU core? icedfusion Linux - Newbie 2 12-31-2008 10:10 AM
gil = TCP/IP process consumes a lot of CPU pete83 AIX 1 05-09-2008 02:57 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Networking

All times are GMT -5. The time now is 09:52 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration