Control of CPU cores utilization on centos server, is it possible?
 

I have server with 4 core processor and I'm using centos (v5.8 now, latest kernel, upgraded yesterday).

Before, one process could use only one core to 100%, which left other 3 cores available to other processes. But after I upgraded my server (I didn't upgraded for a very long time) with yum upgrade, situation is different now.

Now one process could use all cores to the max, which causes problems with other processes and server becomes unresponsive till that one process finishes and leave processor available to other processes.

It is also visible with htop, before each core load was more or less independent, but now they act "like one", i.e. share more or less same load.

This has a good side, as process finishes sooner, but all in all, as server needs to be multitasking environment it impacts it more negatively in my opinion.

Is there a way that I could setup core utilization, for example saying that one process could use only one (or two, or more) core? Any advice how to control this?

This may help you.

http://www.cyberciti.biz/tips/settin...r-process.html

This is what cgroups (containers) is all about - well, not just CPUs.
Define a "mask" of allowed CPUs and drop your task(s) in that group. I use it to isolate heavy workloads under benchmark testing. Leaves me some CPU(s) to run monitors in.

Not well documented last I looked - here is the doco from the kernel source tree. Have a look at cpusets.txt (what it started out as).

I read both links, but I must say that it looks too complicated for my knowledge now.

Isn't there some configuration file with some settings to do that? I want all processes (running and future ones) to use only one core and I don't know what PIDs they will have.

OK, there's also one problem I have with this setup that really annoys me.

I'm not sure if it is how it is intended, but as core utilization is now, it seriously affects my server resources. For example, page generation that should be between 1.1-1.3s is now in range of 2.7-3.3s. My server obviously works "harder" to produce results. This is pretty obvious on server load averages also.

But, what is interesting, after I make upgrade with yum, for some time everything works like it worked before, i.e. I see spiking of one core, other cores are unaffected (or affected as they should be, it is live server). This time is short, but page generation is back to normal, server loads are much lower and after some short time everything is back to using all cores.

This is not only annoying, this affects server resources, server speed.

Anyway, I really don't know if it is some process that gets started after some time after upgrade, or something that gets started for I don't know what reason, but I notice that server is for short time in state I'd like to have it.

If anyone has an idea what's going on and what could cause this, or something, please help me.

Rather than just answer your original question, I guess we should have asked for some evidence.

I would think it (extremely) unlikely that any of your "normal" processes suddenly would start using cores like that simply after a kernel update. And it's even more unlikely they'd all start doing it.
Leaving aside the possibility of having been compromised, I would be looking at something like a dodgy driver. That probably means a high proportion of %sy time in your samples (top is probably better at showing this, in htop it'll be red bars). Keep an eye on /proc/interrupts - most interrupts shouldn't be driving all CPUs, there is usually one or two that have a (much) higher count.
Also high context switching is a bit of a giveaway.

As for tracking which driver is (possibly) the culprit, that is really tough. But it always seems to be the network drivers.

Well, I really don't know what and why is this, but I just notice this.

Thing is, as you mention, in htop I see a bit more of red in cores, but it is not all red. So, there's a lot of user load, it's not all system load. And, I didn't touched drivers, I just did yum upgrade and after that this started. OK, I did upgraded Plesk also (from 10.1.1 to 10.4.4), this could also have something to do with this. But, I'm not sure. :(

Each process or just some applications? Maybe for some applications now parallel versions were installed, which use all cores by OpenMP or as plain threads.

As far as I can tell, this happens for everything, or at least it happens with Apache webserver for sure, as this is most used application I have. Also, mysqldump uses all cores (when I do manual DB backup). So, I could assume that (almost) everything uses all cores. But, it is hard to tell...

Apache can start additional processes, so I would suggest to check the Apache documentation about their tuning parameters. But are you concerned that a process uses to many resources? If you have idle cores around, it’s advantageous to use them.

The kernel is well written to be multi-core enabled by default, BUT its still the responsibility of the App to be written to be multi-tasking or multi-threading to take advantage, & in some cases this wouldn't make sense anyway.

In theory you can use http://linux.die.net/man/1/taskset, but generally I'd let the kernel handle it.

Yes, of course that this sounds good and I too thought that it will help my server run faster. But as it turns out, my server runs slower. For example, when I run mysqldump for manual backup of DB, it takes all cores to 100% leaving my server unresponsive to other task, making my site "frozen" till mysqldump finishes it job.

Also, as I said, watching server load averages and page generation time I see that my server is slower, not faster. Before upgrade, when each core was independent it worked faster.

Something is changed that affects server performance and it is related with upgrade. It maybe something with drivers (as suggested here), or just with settings (I don't know which one), but server now is slower, less efficient.

One option could be to run mysqldump via nice, i.e. with a lower priority.

Well, if it would only be problem related with mysqldump, that would be solution. But I run mysqldump here and there, not all the time. But I have slow down of my server all the time and this is what's bothers me.

I mentioned it just to describe what's happening, not that it really bothers me so much that I need solution for mysqldump "usurping" all cores. Thing is, something is changed after upgrade that seriously degraded my server's performance. I want to solve that problem, if it is possible. And that will solve problem with mysqldump as well.

Just found out what was causing my problem and how to solve it.

I was using Spri (http://www.rfxn.com/projects/system-priority/) and for some reasons, my configuration of spri caused system to behave as I described. When I reseted renice to 0, everything was back to normal. It seems that newer kernels doesn't like how I set renice (very high and low numbers). Now I'm testing to use values from -3 to 2, just to see if it will work as it should. If not, I'll stop using spri altogether.

Anyway, just to inform you what was problem and how to solve it. It might help someone.