LinuxQuestions.org
View the Most Wanted LQ Wiki articles.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Server
User Name
Password
Linux - Server This forum is for the discussion of Linux Software used in a server related context.

Notices

Reply
 
Search this Thread
Old 06-18-2012, 07:01 PM   #1
Alan_SP
LQ Newbie
 
Registered: Mar 2012
Posts: 16

Rep: Reputation: Disabled
Control of CPU cores utilization on centos server, is it possible?


I have server with 4 core processor and I'm using centos (v5.8 now, latest kernel, upgraded yesterday).

Before, one process could use only one core to 100%, which left other 3 cores available to other processes. But after I upgraded my server (I didn't upgraded for a very long time) with yum upgrade, situation is different now.

Now one process could use all cores to the max, which causes problems with other processes and server becomes unresponsive till that one process finishes and leave processor available to other processes.

It is also visible with htop, before each core load was more or less independent, but now they act "like one", i.e. share more or less same load.

This has a good side, as process finishes sooner, but all in all, as server needs to be multitasking environment it impacts it more negatively in my opinion.

Is there a way that I could setup core utilization, for example saying that one process could use only one (or two, or more) core? Any advice how to control this?
 
Old 06-18-2012, 08:37 PM   #2
rosehosting.com
Member
 
Registered: Jun 2012
Posts: 222

Rep: Reputation: 62
This may help you.

http://www.cyberciti.biz/tips/settin...r-process.html


Quote:
Originally Posted by Alan_SP View Post
I have server with 4 core processor and I'm using centos (v5.8 now, latest kernel, upgraded yesterday).

Before, one process could use only one core to 100%, which left other 3 cores available to other processes. But after I upgraded my server (I didn't upgraded for a very long time) with yum upgrade, situation is different now.

Now one process could use all cores to the max, which causes problems with other processes and server becomes unresponsive till that one process finishes and leave processor available to other processes.

It is also visible with htop, before each core load was more or less independent, but now they act "like one", i.e. share more or less same load.

This has a good side, as process finishes sooner, but all in all, as server needs to be multitasking environment it impacts it more negatively in my opinion.

Is there a way that I could setup core utilization, for example saying that one process could use only one (or two, or more) core? Any advice how to control this?
 
Old 06-18-2012, 09:06 PM   #3
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 12,483

Rep: Reputation: 1077Reputation: 1077Reputation: 1077Reputation: 1077Reputation: 1077Reputation: 1077Reputation: 1077Reputation: 1077
This is what cgroups (containers) is all about - well, not just CPUs.
Define a "mask" of allowed CPUs and drop your task(s) in that group. I use it to isolate heavy workloads under benchmark testing. Leaves me some CPU(s) to run monitors in.

Not well documented last I looked - here is the doco from the kernel source tree. Have a look at cpusets.txt (what it started out as).
 
Old 06-19-2012, 08:35 PM   #4
Alan_SP
LQ Newbie
 
Registered: Mar 2012
Posts: 16

Original Poster
Rep: Reputation: Disabled
I read both links, but I must say that it looks too complicated for my knowledge now.

Isn't there some configuration file with some settings to do that? I want all processes (running and future ones) to use only one core and I don't know what PIDs they will have.
 
Old 07-06-2012, 10:00 PM   #5
Alan_SP
LQ Newbie
 
Registered: Mar 2012
Posts: 16

Original Poster
Rep: Reputation: Disabled
OK, there's also one problem I have with this setup that really annoys me.

I'm not sure if it is how it is intended, but as core utilization is now, it seriously affects my server resources. For example, page generation that should be between 1.1-1.3s is now in range of 2.7-3.3s. My server obviously works "harder" to produce results. This is pretty obvious on server load averages also.

But, what is interesting, after I make upgrade with yum, for some time everything works like it worked before, i.e. I see spiking of one core, other cores are unaffected (or affected as they should be, it is live server). This time is short, but page generation is back to normal, server loads are much lower and after some short time everything is back to using all cores.

This is not only annoying, this affects server resources, server speed.

Anyway, I really don't know if it is some process that gets started after some time after upgrade, or something that gets started for I don't know what reason, but I notice that server is for short time in state I'd like to have it.

If anyone has an idea what's going on and what could cause this, or something, please help me.
 
Old 07-06-2012, 11:20 PM   #6
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 12,483

Rep: Reputation: 1077Reputation: 1077Reputation: 1077Reputation: 1077Reputation: 1077Reputation: 1077Reputation: 1077Reputation: 1077
Rather than just answer your original question, I guess we should have asked for some evidence.

I would think it (extremely) unlikely that any of your "normal" processes suddenly would start using cores like that simply after a kernel update. And it's even more unlikely they'd all start doing it.
Leaving aside the possibility of having been compromised, I would be looking at something like a dodgy driver. That probably means a high proportion of %sy time in your samples (top is probably better at showing this, in htop it'll be red bars). Keep an eye on /proc/interrupts - most interrupts shouldn't be driving all CPUs, there is usually one or two that have a (much) higher count.
Also high context switching is a bit of a giveaway.

As for tracking which driver is (possibly) the culprit, that is really tough. But it always seems to be the network drivers.
 
Old 07-08-2012, 08:26 AM   #7
Alan_SP
LQ Newbie
 
Registered: Mar 2012
Posts: 16

Original Poster
Rep: Reputation: Disabled
Well, I really don't know what and why is this, but I just notice this.

Thing is, as you mention, in htop I see a bit more of red in cores, but it is not all red. So, there's a lot of user load, it's not all system load. And, I didn't touched drivers, I just did yum upgrade and after that this started. OK, I did upgraded Plesk also (from 10.1.1 to 10.4.4), this could also have something to do with this. But, I'm not sure.
 
Old 07-09-2012, 03:21 AM   #8
Reuti
Senior Member
 
Registered: Dec 2004
Location: Marburg, Germany
Distribution: openSUSE 13.1
Posts: 1,320

Rep: Reputation: 252Reputation: 252Reputation: 252
Each process or just some applications? Maybe for some applications now parallel versions were installed, which use all cores by OpenMP or as plain threads.
 
Old 07-09-2012, 09:54 AM   #9
Alan_SP
LQ Newbie
 
Registered: Mar 2012
Posts: 16

Original Poster
Rep: Reputation: Disabled
As far as I can tell, this happens for everything, or at least it happens with Apache webserver for sure, as this is most used application I have. Also, mysqldump uses all cores (when I do manual DB backup). So, I could assume that (almost) everything uses all cores. But, it is hard to tell...
 
Old 07-09-2012, 11:08 AM   #10
Reuti
Senior Member
 
Registered: Dec 2004
Location: Marburg, Germany
Distribution: openSUSE 13.1
Posts: 1,320

Rep: Reputation: 252Reputation: 252Reputation: 252
Apache can start additional processes, so I would suggest to check the Apache documentation about their tuning parameters. But are you concerned that a process uses to many resources? If you have idle cores around, it’s advantageous to use them.
 
Old 07-10-2012, 12:58 AM   #11
chrism01
Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.6, Centos 5.10
Posts: 16,324

Rep: Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041
The kernel is well written to be multi-core enabled by default, BUT its still the responsibility of the App to be written to be multi-tasking or multi-threading to take advantage, & in some cases this wouldn't make sense anyway.

In theory you can use http://linux.die.net/man/1/taskset, but generally I'd let the kernel handle it.
 
Old 07-10-2012, 06:43 AM   #12
Alan_SP
LQ Newbie
 
Registered: Mar 2012
Posts: 16

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by Reuti View Post
If you have idle cores around, itís advantageous to use them.
Yes, of course that this sounds good and I too thought that it will help my server run faster. But as it turns out, my server runs slower. For example, when I run mysqldump for manual backup of DB, it takes all cores to 100% leaving my server unresponsive to other task, making my site "frozen" till mysqldump finishes it job.

Also, as I said, watching server load averages and page generation time I see that my server is slower, not faster. Before upgrade, when each core was independent it worked faster.

Something is changed that affects server performance and it is related with upgrade. It maybe something with drivers (as suggested here), or just with settings (I don't know which one), but server now is slower, less efficient.
 
Old 07-10-2012, 06:49 AM   #13
Reuti
Senior Member
 
Registered: Dec 2004
Location: Marburg, Germany
Distribution: openSUSE 13.1
Posts: 1,320

Rep: Reputation: 252Reputation: 252Reputation: 252
One option could be to run mysqldump via nice, i.e. with a lower priority.
 
Old 07-10-2012, 06:55 AM   #14
Alan_SP
LQ Newbie
 
Registered: Mar 2012
Posts: 16

Original Poster
Rep: Reputation: Disabled
Well, if it would only be problem related with mysqldump, that would be solution. But I run mysqldump here and there, not all the time. But I have slow down of my server all the time and this is what's bothers me.

I mentioned it just to describe what's happening, not that it really bothers me so much that I need solution for mysqldump "usurping" all cores. Thing is, something is changed after upgrade that seriously degraded my server's performance. I want to solve that problem, if it is possible. And that will solve problem with mysqldump as well.
 
Old 11-12-2012, 11:12 PM   #15
Alan_SP
LQ Newbie
 
Registered: Mar 2012
Posts: 16

Original Poster
Rep: Reputation: Disabled
Just found out what was causing my problem and how to solve it.

I was using Spri (http://www.rfxn.com/projects/system-priority/) and for some reasons, my configuration of spri caused system to behave as I described. When I reseted renice to 0, everything was back to normal. It seems that newer kernels doesn't like how I set renice (very high and low numbers). Now I'm testing to use values from -3 to 2, just to see if it will work as it should. If not, I'll stop using spri altogether.

Anyway, just to inform you what was problem and how to solve it. It might help someone.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
how to find all processes cpu utilization & the cpu processor in linux? vasanth.v999 Linux - Server 7 01-10-2012 09:43 AM
High CPU utilization on only one CPU out of 4 CPUs makam.sreekanth Linux - Kernel 2 09-03-2009 05:12 AM
Use mrtg to graph cpu utilization by cores KhurramF Linux - Software 0 04-01-2008 02:50 PM
Linux desktop GUI tool for CPU utilization and bandwidth utilization karimasif Linux - Software 4 03-03-2008 02:09 PM
Utilization of all IP's on my dedicated Server (CentOS 4) bilalinamdar Linux - Newbie 9 02-15-2007 12:05 PM


All times are GMT -5. The time now is 06:07 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration