Hello,

I have 2 real time processes which communicate to and fro through out continuously every cycle using non blocking udp sockets on same machine.
Am using SCHED_DEADLINE for both on preempt_rt patched 4.4.1 kernel, x86_64 laptop.

Periods of process are 1 and 2 are 1ms and 5 ms respectively.(control system simulation involving where 1ms process takes feedback from 5ms process and corrects itself throughout).Runtimes are 0.5 ms and 2.5 ms respectively for process 1 and 2.

When I used shared memory as IPC between 2 processes it worked fine. I replaced this with non blocking UDP sockets, process 1 fails (becomes unstable in terms of control system due to lack of coherence between 2 processes). The jitter observed now using clock_gettime(MONOTONIC) is around 40 microseconds for process 1 and 60 microseconds for process 2. When I reduced period of process 1 from 1ms to 980 micro sec it works. I am thinking of reducing this jitter/scheduler latency or any other possibilities to explore to resolve this issue.

So, what are the kernel config options that I can enable along with preemt_rt real time patch (at least the important ones, may not need platform level specific exhaustive) that I can enable/disable for achieving this ( I do use perf for profiling, sysfs uprobes also so I want to avoid disabling relevant kernel config options required for them to work. User space profiling is enough for me, no need of kernel debugging and profiling).

Thanks and regards,
Bhargav

I have modified some parameters. Sorry I have not given any clue of my existing configuration but I am confused which parameters to give and many things come into picture if I have to give the details of existing configuration.

I have taken my current config file and I see improvement in response of my system (2 processes). The modifications given below:

Modified Timer frequency to 1000HZ (earlier it was 250 HZ)
Disabled "Kernel debugger" in Kernel Hacking
Disabled "Debug x86 FPU code" in Kernel Hacking
Disabled "Collect Scheduler stats" in Kernel Hacking
Disabled "Compile Kernel with debug info" in "Compile time Checks" in "Kernel Hacking"
Disabled "Kprobes" in General Setup

Note: hrtimer has always been enabled.

Now, with above modification and kernel rebuild, I am not required to reduce the period of my process 1 from 1ms to 980 micro sec to work properly as I mentioned in my previous post. Its response is fine with the period of 1ms.

But, when I measured execution time (runtime) of each cycle, and period of execution of each cycle, it turned out there are spikes intermittently when I plotted. I believe (I am analyzing the output, using profiling etc) are due to intermittent scheduler switches. Both my processes are running as SCHED_DEADLINE processes but the general maintenance daemons may be hindering the real time behavior. I am using Ubuntu 16.04, Linux kernel 4.4.0-21, preempt_rt patched. Inputs regarding profiling or understanding the reasons behind this behavior are appreciable. I am trying to check /proc/sched_debug and /sys/kernel/debug/tracing/trace but I am not able to get any conclusive insights (I am looking them first time too so still trying to make out the data) especially due to those log data not containing enough time length of data throughout the process execution time but some fixed minor timeline < 1sec... at the time of opening those files, how can I increase the file logging length duration?

Thanks and regards,
Bhargav

A rt kernel should have everything disabled that creates extra work for the kernel. You've already disabled some of those. You might want to change the scheduler from cfq to some other option, the memory control from slab to slub, disable all the memory compression mechanisms, disable the memory swapping page file, disable most of the settings used to optimize the kernel for server use, set for no preemption,

Disable everything related to security, every networking item except what you really need, every hardware interface you don't have or don't use, and every other setting you don't explicitly require. Most of the kernel config settings are discrete, modular hardware drivers.

If your machine doesn't have matching hardware, it won't load the modules. But ideally, a rt kernel doesn't need to load any modules. Everything is built into the kernel. On an industrial control system it is easier to determine exactly what you need in a kernel, because these systems require far fewer capabilities than other types of computers running Linux.

1 members found this post helpful.

@AwesomeMachine, That was very much helpful. Trying out with changes mentioned. I could not identify/clearly understand the options you have mentioned, but I could basically understand what changes are required to be done...

I have just for now tried below:
-swapoff -a (I have anyways used mlockall(CURRENT|FUTURE), but still I wanted to check this way once).
-unmount debugfs (not sure, but thought this is kind of logging mechanism and would reduce the performance, so wanted to give a try)
-Run my real time processes.

With above things, I see very clear and considerable difference in jitter, in plots of period of runtime process.Still the spikes are present though.. Will check again with rebuilt kernel.

@ AwesomeMachine, There is a great improvement now after rebuilding as per your inputs, the jitter (when I set my thread's period as 1 ms with sched_deadline) is in the order of less than 10micro sec, but still there are spikes observed intermittently through out.


void func()
{
/* some code here */

attr.sched_runtime = 50 * 10 * 1000;
attr.sched_deadline = 50 * 10 * 1000;
attr.sched_period = 100 * 10 * 1000;

attr.size = sizeof (attr);

attr.sched_flags = 0;
attr.sched_nice = 0;
attr.sched_priority = 0;

ret = sched_setattr(0, &attr, flags);

/* some more code here */

while(1)
{
/* realtime processing code here */
sched_yield();
/* time_elapsed() returns time diff (in ms) between previous and current invocation of time_elapsed() */
cycle_period_time_taken = time_elapsed();
/* cycle_period_time_taken is plotted across time to check jitter*/
time_elapsed();

/* Data produce to be consumed by other process */

/* while break condition code */
}



I hope, it is valid and proper way to check jitter/scheduler latency in case inherent, by plotting cycle_period_time_taken variable with time.

Thank you very much...
Bhargav