LinuxQuestions.org
Visit Jeremy's Blog.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software > Linux - Kernel
User Name
Password
Linux - Kernel This forum is for all discussion relating to the Linux kernel.

Notices


Reply
  Search this Thread
Old 12-01-2017, 07:08 AM   #1
barz_83_LQ
LQ Newbie
 
Registered: Apr 2017
Posts: 17

Rep: Reputation: Disabled
Kernel build with PREEMPT_RT patch (Ubuntu 16.04, 4.4.1 kernel) : Inputs related to reducing jitter/scheduler latency


Hello,

I have 2 real time processes which communicate to and fro through out continuously every cycle using non blocking udp sockets on same machine.
Am using SCHED_DEADLINE for both on preempt_rt patched 4.4.1 kernel, x86_64 laptop.

Periods of process are 1 and 2 are 1ms and 5 ms respectively.(control system simulation involving where 1ms process takes feedback from 5ms process and corrects itself throughout).Runtimes are 0.5 ms and 2.5 ms respectively for process 1 and 2.

When I used shared memory as IPC between 2 processes it worked fine. I replaced this with non blocking UDP sockets, process 1 fails (becomes unstable in terms of control system due to lack of coherence between 2 processes). The jitter observed now using clock_gettime(MONOTONIC) is around 40 microseconds for process 1 and 60 microseconds for process 2. When I reduced period of process 1 from 1ms to 980 micro sec it works. I am thinking of reducing this jitter/scheduler latency or any other possibilities to explore to resolve this issue.

So, what are the kernel config options that I can enable along with preemt_rt real time patch (at least the important ones, may not need platform level specific exhaustive) that I can enable/disable for achieving this ( I do use perf for profiling, sysfs uprobes also so I want to avoid disabling relevant kernel config options required for them to work. User space profiling is enough for me, no need of kernel debugging and profiling).

Thanks and regards,
Bhargav
 
Old 12-03-2017, 08:51 AM   #2
barz_83_LQ
LQ Newbie
 
Registered: Apr 2017
Posts: 17

Original Poster
Rep: Reputation: Disabled
I have modified some parameters. Sorry I have not given any clue of my existing configuration but I am confused which parameters to give and many things come into picture if I have to give the details of existing configuration.

I have taken my current config file and I see improvement in response of my system (2 processes). The modifications given below:

Modified Timer frequency to 1000HZ (earlier it was 250 HZ)
Disabled "Kernel debugger" in Kernel Hacking
Disabled "Debug x86 FPU code" in Kernel Hacking
Disabled "Collect Scheduler stats" in Kernel Hacking
Disabled "Compile Kernel with debug info" in "Compile time Checks" in "Kernel Hacking"
Disabled "Kprobes" in General Setup

Note: hrtimer has always been enabled.

Now, with above modification and kernel rebuild, I am not required to reduce the period of my process 1 from 1ms to 980 micro sec to work properly as I mentioned in my previous post. Its response is fine with the period of 1ms.

But, when I measured execution time (runtime) of each cycle, and period of execution of each cycle, it turned out there are spikes intermittently when I plotted. I believe (I am analyzing the output, using profiling etc) are due to intermittent scheduler switches. Both my processes are running as SCHED_DEADLINE processes but the general maintenance daemons may be hindering the real time behavior. I am using Ubuntu 16.04, Linux kernel 4.4.0-21, preempt_rt patched. Inputs regarding profiling or understanding the reasons behind this behavior are appreciable. I am trying to check /proc/sched_debug and /sys/kernel/debug/tracing/trace but I am not able to get any conclusive insights (I am looking them first time too so still trying to make out the data) especially due to those log data not containing enough time length of data throughout the process execution time but some fixed minor timeline < 1sec... at the time of opening those files, how can I increase the file logging length duration?

Thanks and regards,
Bhargav
 
Old 12-03-2017, 07:46 PM   #3
AwesomeMachine
Senior Member
 
Registered: Jan 2005
Location: USA and Italy
Distribution: Debian testing/sid; OpenSuSE; Fedora; Mint
Posts: 3,291

Rep: Reputation: 587Reputation: 587Reputation: 587Reputation: 587Reputation: 587Reputation: 587
A rt kernel should have everything disabled that creates extra work for the kernel. You've already disabled some of those. You might want to change the scheduler from cfq to some other option, the memory control from slab to slub, disable all the memory compression mechanisms, disable the memory swapping page file, disable most of the settings used to optimize the kernel for server use, set for no preemption,

Disable everything related to security, every networking item except what you really need, every hardware interface you don't have or don't use, and every other setting you don't explicitly require. Most of the kernel config settings are discrete, modular hardware drivers.

If your machine doesn't have matching hardware, it won't load the modules. But ideally, a rt kernel doesn't need to load any modules. Everything is built into the kernel. On an industrial control system it is easier to determine exactly what you need in a kernel, because these systems require far fewer capabilities than other types of computers running Linux.
 
1 members found this post helpful.
Old 12-04-2017, 08:52 AM   #4
barz_83_LQ
LQ Newbie
 
Registered: Apr 2017
Posts: 17

Original Poster
Rep: Reputation: Disabled
@AwesomeMachine, That was very much helpful. Trying out with changes mentioned. I could not identify/clearly understand the options you have mentioned, but I could basically understand what changes are required to be done...

I have just for now tried below:
-swapoff -a (I have anyways used mlockall(CURRENT|FUTURE), but still I wanted to check this way once).
-unmount debugfs (not sure, but thought this is kind of logging mechanism and would reduce the performance, so wanted to give a try)
-Run my real time processes.

With above things, I see very clear and considerable difference in jitter, in plots of period of runtime process.Still the spikes are present though.. Will check again with rebuilt kernel.
 
Old 12-04-2017, 08:37 PM   #5
barz_83_LQ
LQ Newbie
 
Registered: Apr 2017
Posts: 17

Original Poster
Rep: Reputation: Disabled
@ AwesomeMachine, There is a great improvement now after rebuilding as per your inputs, the jitter (when I set my thread's period as 1 ms with sched_deadline) is in the order of less than 10micro sec, but still there are spikes observed intermittently through out.


void func()
{
/* some code here */

attr.sched_runtime = 50 * 10 * 1000;
attr.sched_deadline = 50 * 10 * 1000;
attr.sched_period = 100 * 10 * 1000;

attr.size = sizeof (attr);

attr.sched_flags = 0;
attr.sched_nice = 0;
attr.sched_priority = 0;

ret = sched_setattr(0, &attr, flags);

/* some more code here */

while(1)
{
/* realtime processing code here */
sched_yield();
/* time_elapsed() returns time diff (in ms) between previous and current invocation of time_elapsed() */
cycle_period_time_taken = time_elapsed();
/* cycle_period_time_taken is plotted across time to check jitter*/
time_elapsed();

/* Data produce to be consumed by other process */

/* while break condition code */
}



I hope, it is valid and proper way to check jitter/scheduler latency in case inherent, by plotting cycle_period_time_taken variable with time.

Thank you very much...
Bhargav

Last edited by barz_83_LQ; 12-04-2017 at 08:56 PM.
 
  


Reply

Tags
build from source, config, latency, preempt_rt, scheduler


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
What paremeters to add to .config to allow kernel build with a reiser4 patch [reiser4-for-4.11.0.patch.gz]? Glenn D. Linux - Software 1 08-06-2017 12:49 AM
How to patch and build rpm of a kernel module (btusb.ko) ghoshdibyajyoti Linux - Newbie 1 01-09-2012 10:20 AM
Where to Find a Guide to Realtime Kernel for Low Latency Audio in Ubuntu? dahouse Ubuntu 1 04-23-2007 11:21 PM
kernel patch to build with intel compiler galad Linux - Software 10 02-10-2004 07:08 PM
The linux kernel ...to patch it or to build it that is the question Rodcl Linux - Software 3 07-13-2002 09:19 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Software > Linux - Kernel

All times are GMT -5. The time now is 06:50 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration