LinuxQuestions.org
Review your favorite Linux distribution.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software > Linux - Kernel
User Name
Password
Linux - Kernel This forum is for all discussion relating to the Linux kernel.

Notices

Reply
 
Search this Thread
Old 03-12-2013, 07:51 AM   #1
kernelexplorer
LQ Newbie
 
Registered: Mar 2013
Posts: 1

Rep: Reputation: Disabled
latency spikes with SCHED_RR (rhel 6.3)


Recently a crucial low latency application was moved to a new server and it is suffering from sporadic huge latency spikes. We think it might be caused by some change in behavior of SCHED_RR in the newer kernel and hope you might help us figure this out.

Here are the details


Old server 2.6.32-279.2.1.el6.x86_64 - 8 physical CPUs New server - 2.6.32-279.19.1.el6.x86_64 - 16 physical CPUs
Redhat 6.3

The application has 15 threads. 4 threads are always spinning/polling I/O or Network and 1 thread is blocking but usually active. The remaining 10 threads are blocking and not very active. In the old server we would see consistent cpu utilization of 504% with low (sub-millisecond) latency in the 5 very active threads. In the new server we see that cpu utilization sporadically drops to 400% from 500% correlated to latency spikes of more than 5 seconds for the blocking thread and two of the spinning threads.

This bad behavior occurs when we run the process with all its threads SCHED_RR priority 50; we used the following commands to give the threads said priority (here PIDS is the list of every thread's PID in the process)

for pid in $PIDS; do
ionice -c 1 -n 7 -p $pid
chrt -r -p 50 $pid
done

On the other hand, if on the new server we instead use SCHED_OTHER with -20 niceness

for pid in $PIDS; do
ionice -c 1 -n 7 -p $pid
renice -20 -p $pid
done

the application works well again giving us performance consistent with the old server SCHED_RR setup. However, this configuration does not give us the same realtime guarantees we want that we believe SCHED_RR is meant to provide. Note the old server even has half as many physical CPUs! We also run this application with 2.6.32-279.2.1.el6.x86_64 SCHED_RR on a 24 core machine with excellent results. Only the 2.6.32-279.19.1 server is giving us trouble with SCHED_RR.

Are you aware of any differences between 2.6.32-279.2.1 and 2.6.32-279.19.1 that could explain or even relate to this behavior? Do you have any other thoughts on what might be causing this difference in behavior?
 
Old 03-13-2013, 09:10 AM   #2
Themace
LQ Newbie
 
Registered: Mar 2013
Posts: 1

Rep: Reputation: Disabled
I'm not sure if this will help but here are the kernel diffs that seem relevant to scheduling. There do seem to be a few subtle scheduling changes between those versions.

Code:
diff -r linux-2.6.32-279.2.1.el6.x86_64/kernel/sched.c linux-2.6.32-279.19.1.el6.x86_64/kernel/sched.c
814a815,817
> #ifndef __GENKSYMS__
> 	struct task_struct *stop;
> #endif
1988a1992
> #include "sched_stoptask.c"
1994c1998,2028
< #define sched_class_highest (&rt_sched_class)
---
> void sched_set_stop_task(int cpu, struct task_struct *stop)
> {
> 	struct sched_param param = { .sched_priority = MAX_RT_PRIO - 1 };
> 	struct task_struct *old_stop = cpu_rq(cpu)->stop;
> 
> 	if (stop) {
> 		/*
> 		 * Make it appear like a SCHED_FIFO task, its something
> 		 * userspace knows about and won't get confused about.
> 		 *
> 		 * Also, it will make PI more or less work without too
> 		 * much confusion -- but then, stop work should not
> 		 * rely on PI working anyway.
> 		 */
> 		sched_setscheduler_nocheck(stop, SCHED_FIFO, &param);
> 
> 		stop->sched_class = &stop_sched_class;
> 	}
> 
> 	cpu_rq(cpu)->stop = stop;
> 
> 	if (old_stop) {
> 		/*
> 		 * Reset it back to a normal scheduling class so that
> 		 * it can die in pieces.
> 		 */
> 		old_stop->sched_class = &rt_sched_class;
> 	}
> }
> 
> #define sched_class_highest (&stop_sched_class)
5720a5755,5768
> static cputime_t scale_utime(cputime_t utime, cputime_t rtime, cputime_t total)
> {
> 	u64 temp = rtime;
> 
> 	temp *= utime;
> 
> 	if (sizeof(cputime_t) == 4)
> 		temp = div_u64(temp, (u32) total);
> 	else
> 		temp = div64_u64(temp, (u64) total);
> 
> 	return (cputime_t) temp;
> }
> 
5730,5736c5778,5780
< 	if (total) {
< 		u64 temp = rtime;
< 
< 		temp *= utime;
< 		do_div(temp, total);
< 		utime = (cputime_t)temp;
< 	} else
---
> 	if (total)
> 		utime = scale_utime(utime, rtime, total);
> 	else
5763,5769c5807,5809
< 	if (total) {
< 		u64 temp = rtime;
< 
< 		temp *= cputime.utime;
< 		do_div(temp, total);
< 		utime = (cputime_t)temp;
< 	} else
---
> 	if (total)
> 		utime = scale_utime(cputime.utime, rtime, total);
> 	else
5940,5941c5980
< 	class = sched_class_highest;
< 	for ( ; ; ) {
---
> 	for_each_class(class) {
5945,5949d5983
< 		/*
< 		 * Will never be NULL as the idle class always
< 		 * returns a non-NULL p:
< 		 */
< 		class = class->next;
5950a5985,5986
> 
> 	BUG(); /* the idle class will always have a runnable task */
6892a6929,6938
> 
> 	/*
> 	 * Changing the policy of the stop threads its a very bad idea
> 	 */
> 	if (p == rq->stop) {
> 		__task_rq_unlock(rq);
> 		spin_unlock_irqrestore(&p->pi_lock, flags);
> 		return -EINVAL;
> 	}
> 
diff -r linux-2.6.32-279.2.1.el6.x86_64/kernel/sched_fair.c linux-2.6.32-279.19.1.el6.x86_64/kernel/sched_fair.c
1135a1136,1138
> 		if (delta < 0)
> 			return;
> 
diff -r linux-2.6.32-279.2.1.el6.x86_64/kernel/sched_rt.c linux-2.6.32-279.19.1.el6.x86_64/kernel/sched_rt.c
1104c1104
< 	if (rt_rq_throttled(rt_rq))
---
> 	if (rt_rq_throttled(rt_rq) && !task_has_rt_policy(rq->idle))
Only in linux-2.6.32-279.19.1.el6.x86_64/kernel: sched_stoptask.c
diff -r linux-2.6.32-279.2.1.el6.x86_64/kernel/stop_machine.c linux-2.6.32-279.19.1.el6.x86_64/kernel/stop_machine.c
297a298,299
> extern void sched_set_stop_task(int cpu, struct task_struct *stop);
> 
302d303
< 	struct sched_param param = { .sched_priority = MAX_RT_PRIO - 1 };
315d315
< 		sched_setscheduler_nocheck(p, SCHED_FIFO, &param);
316a317,318
> 		kthread_bind(p, cpu);
> 		sched_set_stop_task(cpu, p);
321d322
< 		kthread_bind(stopper->thread, cpu);
335a337
> 		sched_set_stop_task(cpu, NULL);
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Sample program that uses SCHED_RR and SCHED_FIFO famsinyi Programming 3 03-11-2013 09:13 AM
causes of CPU spikes Boerboel_Boy Linux - Hardware 4 03-18-2011 02:45 AM
Syslog Cause Spikes? cdegroat Linux - General 1 04-21-2005 11:53 AM
events/0 spikes Moloko Debian 5 02-03-2005 08:08 AM
ReiserFS Spikes JJX Linux - General 0 12-23-2004 04:46 AM


All times are GMT -5. The time now is 05:02 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration