Hi All,
I've a centOS server which I'm using from a dedicated host provided which I manage by myself.

I host around 100 websites most of them are wordpress.
Traffic is not too high. Overall the max traffic for a month is not more than 100k page views.

My server has been crashing lately, either it was overloaded with too many httpd processes or the mqsql is gone.

I've installed a script to monitor and send me e-mail when server loads shoots beyond 2. But id doesn't help. All I can see is more than 20-30 httpd processes running at any given point of time.

I've no clue what is going on. I've optimized all wordpress sites by installing WP Super Cache plugin, Drupal sites are optimized too and other cleanups. But no luck.

All of scripts are upgraded to the latest versions.
I don't have any Joomla installed so it rules out Joomla. No CRE Loaded or shopping carts.

It still crashes on regular basis.

Can anyone suggest how do I find out what is causing problem and fix this issue?

ps -ef | grep http | wc -l
gives
33

Hi
wordpress is an enemy of servers!
It is almost certain that servers hosting wordpress sites have troubles.
There are two ways to obtain solution, one is.

- optimize wordpress with plugins as supercache
- optimize php with PHP accelerator
- optimize http and mysql configuration

Other way, after optimization above listed, you have to use two servers, one for database and one for http processes, but it is extreme solution

Are you talking a kernel panic? Have you checked the logs?

Have you made any changes to the server such as adding sites, Nics, Hard Drives... What was the last change you made before you started having trouble.

How much ram / swap? What percentage is in use. How much Hard Drive space? Any partitions full? My initial guess is you are running out of RAM or a partition is full / filling up (say /tmp). Is Swap a partition or a file? Is /tmp a directory or partition?

20 - 30 http processes doesn't seem excessive but you can set a limit on the number of processes http can spawn.

Tx for reply.
WP Supercache Pluging is already installed in almost all heavy traffic wordpress instalations but it doesn't help.

I've to check with "optimize php with PHP accelerator"

Could you explain on "optimize http and mysql configuration"

Tx

RAM is 4GB, Hard Disk 2x 750 GB
Partitions are not full
I can't add more RAM as host doesn't allow to add more RAM.
I didn't made any major changes, it has been like this since I bought the server. Since beginning I've been installing Wordpress based blogs on regular basis.

I've another server with exactly similar configuration but with 3 wordpress installations and 2/3 Drupal which never crashed since I bought it.

So in theory Wordpress might be a culprit. However I still feel that something else is also going on.

I'm not expert in Linux, I've even brought in 2 experts but they can't fix it either. Both of them say to add more RAM which is not possible.

1) So lets come to RAM problem. How do I check/monitor/log RAM issues? Any help?
2) I've seen sometimes /tmp is full, so how do I fix it
Here are my partitions
df -k
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/md1 20318408 12568760 6725652 66% /
/dev/md2 463694888 72008164 368317872 17% /home
/dev/shm 1022876 0 1022876 0% /dev/shm
/usr/tmpDSK 2015824 41036 1872388 3% /tmp

Swap is a file, is this a problem?

I create one mysql database per wordpress installation, so effectively I've about 100 of mysql databases.

Is this a bad practice? Does it affect performance?
Should I create fewer mysql databases?

every wordpress installation has its database, so no problem.
Perhaps you make confusion between instances and databases.

For mysql optimization i mean to enable query cache, for example
For http server, if you use apache, i mean to check some configurations, such as

Timeout
KeepAlive
MaxKeepAliveRequests
KeepAliveTimeout
MaxClients
MaxRequestsPerChild

For wp optimitazion you can use many howto, such as
http://elliottback.com/wp/why-my-wor...er-than-yours/
http://linuxstuffs.wordpress.com/mysql-optimization/

It should be important to understand is it is a particular blog which can make problem.....

I mean for each wordpress installation I create one mysql Database. So if I've 100 wordpress installations (for 100 domains) I'll have 100 mysql databases on the server.
Is this practice ok?

Another alternative is to create one database and make all 100 wordrpess initializations to use that database but with different table names.

Tx these links have some amazing tips. I'm going to try them soon.

Yes, this pratcice is ok

Swap as a file is not best practice (in my opinion), but in all honesty, unless your file system is full, I doubt that is the problem. How large is your swap file? You may need to increase it's size.

/tmp filling up can be a problem. How to fix it is a good question. Obviously, you could increase the size of /tmp but I am not sure that is the right answer. You need to find out where these tmp files are coming from, if they are filling the partition that may lead you to solution, or at least help identify the problem.

Do the log files provide any clues?
To monitor memory usage look at tools like:
top
vmstat
cat /proc/meminfo

top is really useful for watching in near real time what the system is working on and what your resources are.

The other two can be used at the cli or via cron to take a peak at things over time. vmstat will give a short status if things and cat /proc/meminfo will tell you more about memory usage than you ever wanted to know.

How often is the server crashing? Is it a Kernel Panic or are specific processes (such as mysql or apache) dying or freezing?

Does it happen at a specific time of day or is it random?

The logs should also shed some light if it is a specific WordPress site, script, or ??? causing the problem, so be sure to check the logs thoroughly.


Finally keep in mind, when I asked about changes to your server, that would include things like adding a new Wordpress site, changes to the configuration of java, even when updates are applied. It may seem quite mundane and not worth tracking, but knowing every little change (preferably written in a change log) helps isolate problems, such as these, that crop up seemingly out of nowhere and are hard to track down.

Hi NSN,
Tx for reply.
Yes I alredy have a monitoring script running in cron which dumps the output of above commands and also sends me e-mail if loads goes above 2.0

I looked log and found followings


================
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r b swpd free buff cache si so bi bo in cs us sy id wa st
0 7 93776 53164 124380 1325396 1 1 419 128 282 125 9 8 75 8 0

USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
mysql 5324 0.5 17.6 450116 360148 ? Sl Feb02 3:24 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/server.net.pid --skip-external-locking
account1 17303 0.0 1.5 113196 32128 ? R 17:12 0:00 /usr/bin/php /home/account1/public_html/index.php
account2 30731 11.0 0.0 0 0 ? Z 01:41 0:00 [php] <defunct>
account3 30735 16.0 0.0 0 0 ? Z 01:41 0:00 [php] <defunct>
=============================


Observation
1) I see that mysqld is taking a lot of memory and it is pretty normal in all log snaps.
Any idea what is that?


2) Then there are the Wordpress blogs account1/2/3 which takes up a lot of CPU.
These wordpress blogs either show the /public_html/index.php which is for home page or for admin page /public_html/wp-admin/index.php

I'm not sure why Wordpress Admin takes so much resources? I've only standard Wordpress plugins installed such as Wordpress.com Stats, Akismet, All in SEO.

The only plugin which might be problematic is YARPP but still can't blame it as WP Super Cache is installed and developer of this plugin says that with cache it should not cause problem.

So basically that is it.
Wordpress php is causing problem.

I've increased the swap file size to be 2 GB, seems to little more stable though today.

The crash happens almost everyday now, earaly morning UK time.

I'm yet to analyze the logs.

Any clue with above ?

Just some quick thoughts:
- Attach (or pastebin?) crash log(s) as plain text file,
- Run your (historical!) system and daemon logs through Logwatch for a quick glance quick win overview of issues,
- Review what changes were made to the system from say one month prior to the crashes.

Hi Jack,

I think the 2 Gb swap is definitely not enough in your case. Take a look this recommendation here: http://www.walkernews.net/2010/05/02...ecommendation/

All the time I used phisical memory + 100-200Mb RAM for swap. No matters Linux or W$ndows op.

All the files that used by Mysql and the httpd server uses the cache memory and swapping.
If the tmp directory full that can cause mysql server to die easily too.

The wordpress opens all the sites that is used by the users and that sites kept in the memory even with out the super cache plugin. (The op system does that)
I'm surprised that the gurus didn't find the swap file tooooo small.

Anyway make a 4.5Gb swap file and try to clean up the tmp directory too.

You said something about the latest packages.
Well the server that I use right now can't be upgraded to the latest word press because after the upgrade the admin part does not show anything at all.
I guess the Mysql and the php hasn't been upgraded to the latest packages so the latest wordpress can't be used at all.
I tried it twice so I'm sure.

So try to find something about the latest wordpress bugs with php/mysql problems.
I would not be surprised if you would get a lots of hit about that bugs.

Oh and something come up in my mind.
Try to increase the maximum open files under the system.

Here is a link how to do that in CentOS: http://www.cyberciti.biz/faq/linux-i...of-open-files/
If the max open files reached that can cause to die the mysql server.

Laz

Tx Mate
I'll try to play around with the changes you suggested.

I've already implemented the Apache settings.

And just to test I've deactivated YARPP, Wordpress.com Stats plugins and monitoring this blog.
I'm sure the problem is from Wordpress which is getting multiplied by 100.

Let's see how it goes.