Hello. I'm having a 2 CentOs 4.3 boxes that tend to get overloaded every once in a while (get to 100 when the problem occur, when they normally go around the 1.0-2.0). Some tests I've made, pointed out that the main problem is that writing to disk creates overload. A simple cp of a 4.4G directory could demostrate the problem:

[02:59:43][ *root* @ machine:/usr/local]$ w | head -n1
02:59:48 up 16 days, 18:38, 11 users, load average: 0.65, 0.96, 1.05
[02:59:48][ *root* @ machine:/usr/local]$ cp -rf apache_prod/ tmp &
[1] 16694
[03:01:03][ *root* @ machine:/usr/local]$ w | head -n1
03:01:08 up 16 days, 18:39, 11 users, load average: 2.91, 1.48, 1.22
[03:01:09][ *root* @ machine:/usr/local]$ w | head -n1
03:01:58 up 16 days, 18:40, 11 users, load average: 6.01, 2.45, 1.55

In a matter of 2 min, the load climbed up to 6, and it would have gone ferther if I would't have stopped it.

As I said, this is the problem happens on 2 machine, with rather different hardware profile: one has SATA disk, using LVM, the other is using and IDE disk, no LVM. Swap is not overused on both machines, and just watch the hdparm test:
[03:10:52][ *root* @ machine:/usr/local]$ hdparm -Tt /dev/sda
/dev/sda:
Timing cached reads: 1726 MB in 2.00 seconds = 862.70 MB/sec
Timing buffered disk reads: 174 MB in 3.03 seconds = 57.42 MB/sec
(The other machine is quite the same).

Both machine host several sites using apache 2.0.52 and PHP 5.2.0, and mysql 5.0.24a-standard. Nothing I find problematic.

Can anyone assist?
Thanks in advance
Yair

Hi guruyaya,
open another command prompt then start top, check 'wa' status. If % high maintain long time means your harddisk already reach bottle neck. Please check is there any application always read/write into your harddisk(Probably beagle, updatedb or etc).

Recommended solution is separate folder which always read/write (/usr, /var, or etc) into difference hard disk.

Good luck,
Ks

Two words: File Server. Two other good words are RAID Array.

CPU and IO bus bottleneck is one good reason file servers were created to smooth out access on very large files accessed often or many many small files to lessen disk thrashing. The computer is, i suspect, not happy about read/writing 4 gigs on a not very fast HD but most of the time, as you state, its doing its best at it.

For comparison, i can get my cpu load to 35 by doing ls -R from /
So dont worry about it unless you want to get into mission critical redundant failsafe stuff and spend a ton of money.

I'm afraid this is a production machine, that hosts several sites. I'm afraid that I cannot "not worry" about it.

I have, however, understood that the right way is to cut on disk read/write. As I moved the apache logs to the home folder (that makes a lot of read write, and is meant to do that), it got a bit better.
To my surprise, that machine that had an improvement was the one with the LVM, using one physical disk. The other machine, with 2 disks (Home folder is set on another physical disk). This machine hosts more sites, but it is, for some reason, more sensitive. It is holding steady on 0.97-1.5, while I would like to see it on less then one most of the time.
As it seems, I need to see if there's something writing constantly, on this machine to the disk, without my knowledge.

Thanks
Yair

u can write a script to email you every 15 minute server load daily. Then you know whether your server is over load or not. 15 minute x 4 x 24= 1440 records. Finally you can plot a graph with openoffice to see the trend.

The easier way is using MRTG if it exist in your server.

Acctually, I allready did that, but every 2 min. It helps noticing a big problem (like when I'm very high on load avg), but can't say it helps solving it, at least for now.

So the graph indicate your server overloaded? Do you check the I/O graph too? Probably your can configure syslog to write log file into 1 dedicated machine too. Using Raid 5 is a good ideal too (It increase your read/write speed and also give you redundancy) since it is mission critical server.
Good luck,

Ks