Trying to troubleshoot sudden server slowdown
I have a VPS, running Ubuntu 10.04.02 LTS, and ever since I applied some of the security updates several months ago, the server will eventually run out of resources and force me to do a hard reset.
Since it happens so quickly, and once it happens, I can't do anything via the command line, I have not figured out yet what process is causing this. I am guessing it is either Apache or MySQL since there isn't much else running on that server. free -m output: Code:
Mem: 1712 1692 20 0 56 396 |
Have you checked your Apache and MySQL logs for more info? I don't run Ubuntu so I'm not familiar with its file-system layout, but you could check /var/log and /usr/local/apache2/logs as a start.
|
When you say it runs out of resources, do you mean it runs out of ram and starts swapping? Do you mean the disk IO is too high? CPU load spike? What I have seen on quite a few VPS/Cloud Servers is a default apache config with too many MaxSpareServers which end up eating all of the available RAM. This then causes the VPS to swap and its all downhill from there. This is just a blind shot in the dark as it could be countless things that are causing issues for you.
|
I'm pretty sure it is memory or CPU because I did configure webmin to alert me when load values increase rapidly. The problem is, I receive a few e-mails, and by the time I get to the system (usually within a minute), it's in such bad shape that I can't do anything. If I am logged in already, I can try to type a command, but whatever I type appears very slowly, which also makes me believe it is a cpu/memory issue.
|
The good ol' top(1) command gives you a quick look at what's happening WRT swap, CPU usage, and memory usage (among others). I normally go through the following steps to quickly locate an offender:
Hopefully that provides some clues. If not, and/or if this is a regular problem for you, it may be time to install sysstat. |
The problem is that it happens pretty quick, and when it happens, I can't run top or any other commands.
|
Sounds like a fairly normal "no more memory, time to swap" issue. I would work on optimizing apache as it is likely your culprit if you are just running a basic LAMP stack. If you are using WordPress for your sites, I would also recommend installing some caching plugins and maybe xcache for php to lighten the load. In my experience, running a default wp install on your standard 'yum install httpd' apache configs will start eating up memory fast. This is mostly due to the MaxSpareServers settings being too high on a small VPS.
|
Check the amount of space consumed by your log files and make sure logrotate is working correctly.
|
It's a fairly optimized Apache/MySQL install, running Joomla, eAccelerator and various other security components. It's the same configuration that has been running ok for years, but I do need to brush up on my Apache/MySQL performance tuning skills since it has been a while.
Thanks for the sysstat pointer, looks pretty interesting, will check it out. |
I can't believe anyone would run a "critical" server without a logging monitor.
sysstat is good for over-all trends; if you want to be able to pinpoint particular processes over a (much) more granular record, have a look at collectl (in daemon mode). |
I have been running collectl, but am fairly sure that I am dealing with an apache problem, due to memory usage, and restarting apache (when it allows me to) does fix the problem for 24 hours or so.
Code:
root 16835 0.0 7.4 435072 131024 ? Ss 03:47 0:00 /usr/sbin/apache2 -k start Code:
vmstat 10 10 |
So months later, I still haven't figured this out. I started logging the output of vmstat to a log while the server was in good shape, figuring I would catch the output of vmstat as the server dies. Snippet of when the server was running ok:
PHP Code:
PHP Code:
What really does stand out is the memory usage of my apache child processes. It looks pretty excessive: PHP Code:
PHP Code:
Modules loaded: PHP Code:
|
All times are GMT -5. The time now is 01:48 PM. |