HELP: system load on server is very high
My system responds very slow on Apache requests, as far I analyzed top-command it have something to do with high system CPU usage - but how to analyze this further?
Here is the top-output during a standard (small and simple) http request. This runs on another server within 2s, but on the problematic ones it takes minutes: Code:
top - 01:06:10 up 3 days, 9:54, 3 users, load average: 2.02, 1.89, 2.28 The distri ist Debian Squeeze. I have the gut-feeling that the high system-load is somehow caused by Apache, but can't strengthen this thesis. The apache error.log shows nothing... How to find the process causing this?? Edit: The best explanation for high %sy in top I found is: "having higher numbers here may indicate a problem with kernel configs, a driver issue, or any number of other things" here But this don't help... Thanks Achim |
Is the load average consistently high?
Can you post the output of some other performance tools? vmstat, iostat, free etc |
Quote:
Very hard to track down - never looked at embedded. Have a look at /proc/interrupts for hints on what may be playing up. vmstat might indicate abnormal context switches - also a clue that driver/interrupt is the problem. |
Hi all,
the load is not consistently high, it alternates between 2%sy and 90%sy and it looks like it somehow depends from the apache process. If it is restarted, the %sy is very low for a while, after some hours of requests to apache it will come up more and more on each request and therefore the response become much slower - until it is uselsess. A simple apache restart will than bring back the system to be responsive again... so i Guess it have something to do with PHP or Apache. Thanks for the good hint with vmstat, here is some output: The first output was made after Apache processes have been running since several days during a single request which took about 100s: Code:
~>vmstat -a 2 Code:
~> vmstat -a 2 Code:
~>vmstat -s and "iostat" I can't find for my Debian distribution, it seems not to be available, also not in sysstat, but hopefully it's covered in the output above. The /proc/interrupts look OK, the imx-i2c is due to a sensor polled regulary via i²c: Code:
~> cat /proc/interrupts What also is suspicious: I have several of these systems running, same hardware, same processes, hopefully identically installed (never 100% sure, because it is done manually), but this one is so slow by having such high %sy load... Thanks a lot Achim |
As per the top output the load is not high & %sys utilization is very high if it is possible to you to install pkg in that box & get the sar report that will give you all essential date to fetch the problem
# iostat -kdx (disk read write performance check the (service time interval & utilization) # sar -q (give u the load average as per the duration set in cron for the sar logs) more http://www.linuxquestions.org/questi...ck-4175427965/ may be required to fine tune the Apache http://httpd.apache.org/docs/2.2/misc/perf-tuning.html http://www.supportsages.com/blog/201...r-performance/ |
Meanwhile I came to the idea comparing an OK system with the NOK one, because they have really the same hardware, running the same software, only a different machine. Here is an output responding on exactly the same request within 6s, seen in log from line 4 to 6 marked in red:
Code:
~> vmstat -a 2 sar was running successfully, but I don't see the high load there. In the output below the red colored times are during a slow access, see: ~ Code:
> sar -q 2 |
Not sure I would recommend running apache with php on 64M RAM??? Certainly with no swap available!!
|
Hmmm, it is running on some other systems with 64MB well (so at least much faster). If I look to vmstat I see plenty of inactive and free memory...
Usually there is only 1 user logged in, the parameter MaxCients is configured away from default 150 to 4 only, the StartServer is reduced to 2. Apache is what I know best, that's the reason why I started with it also on the small embedded machine (400MHz, 64MB RAM). You are right, maybe it is time now to switch to a more lightweight server like LIGHTTPD. But I have th gut-feelingm that there is another problem laying below, because on other systems it is running well. Maybe with the reduced ressource need of LIGHTTPD the problem is only delayed by some days and at the end being on the same state like now :-( Edit: It seems the problem is solved. By accident I saw three processes in top running with NICE=-6 These self-made processes run regularly and often need top cpu resources. After chenging the back to -1 to be still a little bit better prioritized, everything works fine. Now the %sy is down to about 10 and the system is much more responsive again. |
All times are GMT -5. The time now is 05:28 AM. |