Dear all,

My website is hosted on a vps with 30 GB space and 1GB RAM with Intel(R) Xeon(R) CPU E5620 @ 2.40GHz.

My site platform is drupal.

At times the load averages reach 23 and the site becomes inaccessible. sometimes it becomes stable with restarting of mysql and httpd. But some times it stucks.

thanks

Hello,

As a first step to debug the issue, you can issue top -c command to find which process takes high CPU. You can post the output here.

Thanks a lot for your quick response.
The server is normal at this time. I ll post the output when it will be at higher load. The current output is given below:

17850 mydomain 16 0 43168 26m 5516 R 12.2 2.6 0:00.41 /usr/bin/php /home/mydomain/public_html/index.php
17829 root 18 0 13236 10m 1656 R 5.7 1.0 0:00.24 tailwatchd - chkservd
20387 root 16 0 39504 31m 2652 S 5.4 3.1 0:33.28 spamd child
17784 nobody 16 0 61540 50m 1948 S 3.3 5.0 0:00.19 /usr/local/apache/bin/httpd -k start -DSSL
17729 nobody 15 0 61656 51m 1976 S 0.9 5.0 0:00.12 /usr/local/apache/bin/httpd -k start -DSSL
17649 nobody 15 0 61540 50m 1964 S 0.6 5.0 0:00.22 /usr/local/apache/bin/httpd -k start -DSSL
17823 nobody 15 0 61540 50m 1940 S 0.6 5.0 0:00.02 /usr/local/apache/bin/httpd -k start -DSSL
16368 root 15 0 2684 1128 880 R 0.3 0.1 0:00.35 top -c
1 root 15 0 2896 1380 1200 S 0.0 0.1 0:32.43 init
1165 root 14 -4 2460 564 340 S 0.0 0.1 0:00.00 /sbin/udevd -d
1477 root 18 0 35972 2372 1008 S 0.0 0.2 0:07.09 /sbin/rsyslogd -i /var/run/syslogd.pid -c 5
1515 root 15 0 8636 1032 540 S 0.0 0.1 0:03.00 /usr/sbin/sshd
1522 root 25 0 3216 920 728 S 0.0 0.1 0:00.00 xinetd -stayalive -pidfile /var/run/xinetd.pid
1636 root 18 0 8832 792 348 S 0.0 0.1 0:00.00 /usr/sbin/saslauthd -m /var/run/saslauthd -a pam -n 2
1637 root 18 0 8832 512 68 S 0.0 0.0 0:00.00 /usr/sbin/saslauthd -m /var/run/saslauthd -a pam -n 2
1658 root 15 0 7316 4812 1224 S 0.0 0.5 0:37.08 tailwatchd
1709 root 15 0 8700 1736 1300 S 0.0 0.2 0:00.78 pure-ftpd (SERVER)
1711 root 15 0 9524 1340 1032 S 0.0 0.1 0:00.17 /usr/sbin/pure-authd -s /var/run/ftpd.sock -r /usr/local/cpanel/bin/pureauth
1718 root 18 0 7132 1304 680 S 0.0 0.1 1:27.96 crond
1728 root 18 0 2940 524 380 S 0.0 0.0 0:00.18 /usr/sbin/atd
1852 root 15 0 17584 12m 5000 S 0.0 1.2 0:19.82 cpsrvd (SSL) - waiting for c --llu=1382991725 --listen=3,4,5,6,7,8,9
3610 root 15 0 37992 30m 2728 S 0.0 2.9 0:05.25 /usr/local/cpanel/3rdparty/perl/514/bin/spamd -d --allowed-ips=127.0.0.1 --pidfile=/var/run/spamd.pid --max-children
5625 named 25 0 51116 12m 2600 S 0.0 1.2 0:01.00 /usr/sbin/named -u named
5764 mydomain 16 0 29980 19m 4388 S 0.0 1.9 0:12.54 cpaneld - serving 202.147.17 --llu=1382991725 --listen=3,4,5,6,7,8,9
8083 root 18 0 3040 1320 1132 S 0.0 0.1 0:00.00 /bin/sh /usr/bin/mysqld_safe --datadir=/var/lib/mysql --pid-file=/var/lib/mysql/server.kmu.edu.pk.pid
9238 mysql 15 0 182m 73m 5624 S 0.0 7.2 23:33.77 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --log-error=/var/lib/mysql/server.kmu.edu.pk.err
13606 root 18 0 5908 3472 1444 S 0.0 0.3 0:19.93 queueprocd - wait to process a task
13881 root 15 0 17320 8572 680 S 0.0 0.8 0:02.78 cpdavd - accepting connections on 2077 and 2078
16250 root 15 0 11684 3708 2960 R 0.0 0.4 0:00.10 sshd: root@pts/0
16299 nobody 15 0 61532 50m 1944 S 0.0 5.0 0:00.04 /usr/local/apache/bin/httpd -k start -DSSL
16334 root 15 0 6396 1716 1420 S 0.0 0.2 0:00.01 -bash
17441 nobody 15 0 61540 50m 1968 S 0.0 5.0 0:00.01 /usr/local/apache/bin/httpd -k start -DSSL
17515 mydomain 19 0 0 0 0 Z 0.0 0.0 0:01.28 [php] <defunct>
17615 root 18 0 8360 5900 2116 S 0.0 0.6 0:00.14 /usr/local/cpanel/3rdparty/bin/perl /usr/local/cpanel/bin/leechprotect
17621 root 17 0 61264 48m 136 S 0.0 4.8 0:00.00 /usr/local/apache/bin/httpd -k start -DSSL
17650 nobody 15 0 62052 51m 1972 S 0.0 5.0 0:00.49 /usr/local/apache/bin/httpd -k start -DSSL
17653 nobody 15 0 61728 51m 1968 S 0.0 5.0 0:00.35 /usr/local/apache/bin/httpd -k start -DSSL
17658 nobody 15 0 61540 50m 1948 S 0.0 5.0 0:00.08 /usr/local/apache/bin/httpd -k start -DSSL
17686 nobody 15 0 61540 50m 1964 S 0.0 5.0 0:00.12 /usr/local/apache/bin/httpd -k start -DSSL
17693 nobody 15 0 61540 50m 1964 S 0.0 5.0 0:00.18 /usr/local/apache/bin/httpd -k start -DSSL

The server load is showing 7 at this time and the output of top -c is given below:

26274 kmuedupk 18 0 52776 36m 5852 R 42.1 3.6 0:00.90 /usr/bin/php /home/kmuedupk/public_html/index.php
9238 mysql 15 0 182m 73m 5624 S 3.4 7.1 24:44.39 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --log-error=/var/lib/mysql/server.kmu.edu.pk.err
25831 nobody 15 0 0 0 0 Z 1.7 0.0 0:00.36 [httpd] <defunct>
25841 nobody 16 0 61640 50m 1964 S 1.7 5.0 0:00.33 /usr/local/apache/bin/httpd -k start -DSSL
1 root 15 0 2896 1380 1200 S 0.0 0.1 0:32.43 init
1165 root 14 -4 2460 564 340 S 0.0 0.1 0:00.00 /sbin/udevd -d
1477 root 18 0 35972 2376 1008 S 0.0 0.2 0:07.12 /sbin/rsyslogd -i /var/run/syslogd.pid -c 5
1515 root 15 0 8636 1032 540 S 0.0 0.1 0:03.00 /usr/sbin/sshd
1522 root 25 0 3216 920 728 S 0.0 0.1 0:00.00 xinetd -stayalive -pidfile /var/run/xinetd.pid
1636 root 18 0 8832 792 348 S 0.0 0.1 0:00.00 /usr/sbin/saslauthd -m /var/run/saslauthd -a pam -n 2
1637 root 18 0 8832 512 68 S 0.0 0.0 0:00.00 /usr/sbin/saslauthd -m /var/run/saslauthd -a pam -n 2
1658 root 15 0 7316 4812 1224 S 0.0 0.5 0:37.46 tailwatchd
1709 root 15 0 8700 1736 1300 S 0.0 0.2 0:00.78 pure-ftpd (SERVER)
1711 root 16 0 9524 1340 1032 S 0.0 0.1 0:00.17 /usr/sbin/pure-authd -s /var/run/ftpd.sock -r /usr/local/cpanel/bin/pureauth
1718 root 15 0 7132 1304 680 S 0.0 0.1 1:28.30 crond
1728 root 18 0 2940 524 380 S 0.0 0.0 0:00.18 /usr/sbin/atd
1852 root 15 0 17584 12m 5000 S 0.0 1.2 0:20.02 cpsrvd (SSL) - waiting for c --llu=1382991725 --listen=3,4,5,6,7,8,9
3610 root 15 0 37992 30m 2728 S 0.0 2.9 0:05.43 /usr/local/cpanel/3rdparty/perl/514/bin/spamd -d --allowed-ips=127.0.0.1 --pidfile=/var/run/spamd.pid --max-children
8083 root 18 0 3040 1320 1132 S 0.0 0.1 0:00.00 /bin/sh /usr/bin/mysqld_safe --datadir=/var/lib/mysql --pid-file=/var/lib/mysql/server.kmu.edu.pk.pid
13606 root 18 0 5908 3472 1444 S 0.0 0.3 0:20.02 queueprocd - wait to process a task
13881 root 15 0 17320 8572 680 S 0.0 0.8 0:02.79 cpdavd - accepting connections on 2077 and 2078
17615 root 18 0 8360 5900 2116 S 0.0 0.6 0:00.14 /usr/local/cpanel/3rdparty/bin/perl /usr/local/cpanel/bin/leechprotect
17621 root 17 0 61264 48m 136 S 0.0 4.8 0:00.00 /usr/local/apache/bin/httpd -k start -DSSL
18099 root 33 18 5144 2228 880 S 0.0 0.2 0:00.99 cpanellogd - sleeping for logs
20145 named 25 0 51116 12m 2540 S 0.0 1.2 0:00.03 /usr/sbin/named -u named
20183 mailnull 15 0 11168 1288 748 S 0.0 0.1 0:05.89 /usr/sbin/exim -bd -q1h
20387 root 15 0 39504 31m 2652 S 0.0 3.1 0:35.24 spamd child
24127 root 18 0 13868 8108 1336 S 0.0 0.8 0:08.12 lfd - sleeping
25817 nobody 15 0 61668 51m 1968 S 0.0 5.0 0:00.36 /usr/local/apache/bin/httpd -k start -DSSL
25825 nobody 15 0 61540 50m 1964 S 0.0 5.0 0:00.38 /usr/local/apache/bin/httpd -k start -DSSL
25827 nobody 16 0 61676 51m 1972 S 0.0 5.0 0:00.20 /usr/local/apache/bin/httpd -k start -DSSL
25832 nobody 15 0 61540 50m 1964 S 0.0 5.0 0:00.35 /usr/local/apache/bin/httpd -k start -DSSL
25834 nobody 15 0 61540 50m 1972 S 0.0 5.0 0:00.31 /usr/local/apache/bin/httpd -k start -DSSL
25902 nobody 15 0 61528 50m 1968 S 0.0 5.0 0:00.27 /usr/local/apache/bin/httpd -k start -DSSL
25984 kmuedupk 16 0 0 0 0 Z 0.0 0.0 0:00.72 [php] <defunct>
26039 nobody 15 0 61664 50m 1964 S 0.0 5.0 0:00.08 /usr/local/apache/bin/httpd -k start -DSSL
26045 nobody 15 0 61668 50m 1964 S 0.0 5.0 0:00.08 /usr/local/apache/bin/httpd -k start -DSSL
26051 nobody 15 0 62036 51m 1968 S 0.0 5.0 0:00.18 /usr/local/apache/bin/httpd -k start -DSSL
26052 nobody 15 0 61636 50m 1972 S 0.0 5.0 0:00.10 /usr/local/apache/bin/httpd -k start -DSSL
26053 nobody 15 0 61800 51m 1964 S 0.0 5.0 0:00.08 /usr/local/apache/bin/httpd -k start -DSSL
26059 nobody 15 0 61532 50m 1968 S 0.0 5.0 0:00.15 /usr/local/apache/bin/httpd -k start -DSSL
26075 nobody 16 0 61528 50m 1972 S 0.0 5.0 0:00.09 /usr/local/apache/bin/httpd -k start -DSSL
26080 nobody 15 0 61528 50m 1948 S 0.0 5.0 0:00.12 /usr/local/apache/bin/httpd -k start -DSSL
26097 nobody 15 0 61540 50m 1968 S 0.0 5.0 0:00.09 /usr/local/apache/bin/httpd -k start -DSSL
26103 root 15 0 18352 10m 2772 S 0.0 1.0 0:00.07 whostmgrd - serving 202.147. --llu=1382991725 --listen=3,4,5,6,7,8,9

Looks like the high load is coming from the index.php file. So just check and optimise the code in it.

Actually "-c" (reversed "c" state) doesn't do anything useful unless you have a ~/.toprc?..


Executive overview: do some inventorying, monitor resources, work with what you have or increase resources.

0) You run Drupal so:
- ensure you run the latest version (7.23 right now). Regard updating as mandatory system maintenance.
- check https://drupal.org/documentation and especially https://drupal.org/node/1925512,
- review Drupal functionality and any (3rd party) addons, plugins and whatnot for running the most recent version, right configuration but most of all if you really really need it.

1) More importantly you seem to be running Named, Exim, MysQL, Apache on the same VPS. Each of these services has its own system resource needs. Together with whatever Drupal functionality you enable your VPS may be underspecced to start with:
- ensure you really really need to run a host of web-based management panel processes,
- ensure all configuration is as it should be (like Apache config, php.ini defaults, enabling APC cache or not, etc, etc),
- ensure all configuration changes made reflect your current resource limits (and check what effect changes have),
- monitor system and daemon logs, run Apachetop or equivalent, enable explain / slow query logging,
- monitor system resource statistics (for alerting and restarting services automagically see for example Monit),
- ponder pre-rendering as much as possible, caching and moving certain services to another VPS.

2) More importantly still: don't log in over SSH as root!

HTH

Thanks alot for your detailed answer.
I use WHM and SSH as root user. You have suggested to use them with some other user. What can I understand is to create another user and then user su command.
I want to share some of my configurations and error logs with you, if you allow me. I think there is some problem with my php and httpd configuration.

I am sorry for not following you. I ll try my level best to do so.

thanks

Sure, feel free to post anything relevant. However, as I explained in your other thread, you probably are better of finding or hiring a temporary admin to fix things for you. Because with your skill set and the pace you seem to want to do things this will take ages. I'll emphasize though you're free to post anything relevant that could help solve your problem(s).

Dear, I m trying to learn as quickly as I can. There is no emergency for the time being. But what I want is a permanent solution. In this way I ll also learn from your experience. If I am wasting your time, I am extremely sorry.

Kindly guide me about the errors and how to rectify them by looking into the following log.


root@server [~]# tail -f /usr/local/apache/logs/error_log
[Sun Nov 17 07:28:26 2013] [error] [client 66.249.73.14] (12)Cannot allocate memory: couldn't create child process: /opt/suphp/sbin/suphp for /home/---edupk/public_html/index.php
[Sun Nov 17 07:28:26 2013] [error] [client 66.249.73.14] (12)Cannot allocate memory: couldn't create child process: /opt/suphp/sbin/suphp for /home/---edupk/public_html/index.php
[Sun Nov 17 07:29:57 2013] [error] [client 82.145.220.74] (70007)The timeout specified has expired: ap_content_length_filter: apr_bucket_read() failed
[Sun Nov 17 07:46:34 2013] [error] [client 208.115.111.72] ModSecurity: Rule be42070 [id "390149"][file "/usr/local/apache/conf/modsec_rules/50_asl_rootkits.conf"][line "104"] - Execution error - PCRE limits exceeded (-8): (null). [hostname "---.edu.pk"] [uri "/qec/index.php"] [unique_id "UoguAkj5l5wAAD7kYLAAAAAK"]
[Sun Nov 17 07:57:05 2013] [error] [client 115.167.47.187] ModSecurity: Rule be42070 [id "390149"][file "/usr/local/apache/conf/modsec_rules/50_asl_rootkits.conf"][line "104"] - Execution error - PCRE limits exceeded (-8): (null). [hostname "ipmr.---.edu.pk"] [uri "/index.php"] [unique_id "UogwXkj5l5wAAD-0ekEAAAAL"]
[Sun Nov 17 07:57:10 2013] [error] [client 115.167.47.187] ModSecurity: Rule be42070 [id "390149"][file "/usr/local/apache/conf/modsec_rules/50_asl_rootkits.conf"][line "104"] - Execution error - PCRE limits exceeded (-8): (null). [hostname "ipmr.---.edu.pk"] [uri "/index.php"] [unique_id "Uogwgkj5l5wAAEQPu34AAAAK"]
[Sun Nov 17 08:13:38 2013] [notice] Graceful restart requested, doing restart
[Sun Nov 17 08:13:38 2013] [error] (9)Bad file descriptor: apr_socket_accept: (client socket)
[Sun Nov 17 08:13:40 2013] [notice] SSL FIPS mode disabled
[Sun Nov 17 08:13:41 2013] [notice] Apache/2.2.23 (Unix) mod_ssl/2.2.23 OpenSSL/1.0.0-fips mod_auth_passthrough/2.1 mod_bwlimited/1.4 FrontPage/5.0.2.2635 Apache configured -- resuming normal operations

These are all common errors you can find solutions for searching LQ or using your favorite search engine. Grep your mod_security configuration file, Apache httpd.conf and php.ini for the following:

Like it or not but things are related. So please read my post in which I wrote about assessing the state of the whole system again. A good way to see if you have worked on these issues would be if you could respond per item mentioned there.

Sorry to say that nothing is working for me. I updated drupal to the latest version. But this made the situation more worse for me. Now the load is even more than before.

I have changed memory limits for PHP from 128 to 256 to 512 but to no avail.

I have multiple linnes of /usr/local/apache/bin/httpd -k start -DSSL consuming 5% of the memory each.
I think this is the reason for high load average. How can I reduce it.

whenever I make changes to apache configuration through WHM, following errors are displayed.

[Tue Nov 19 12:58:57 2013] [crit] (EAI 9)Address family for hostname not supported: alloc_listener: failed to set up sockaddr for ::

Configuration problem detected on line 172 of file /usr/local/apache/conf/httpd.conf.work.O60GBbOFSwotmcqu: Listen setup failed

--- /usr/local/apache/conf/httpd.conf.work.O60GBbOFSwotmcqu ---
166# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
167
168
169PidFile logs/httpd.pid
170LockFile logs/accept.lock
171Listen 0.0.0.0:80
172 ===> Listen [::]:80 <===
173User nobody
174Group nobody
175ExtendedStatus On
176ServerAdmin webmaster@kmu.edu.pk
177ServerName server.kmu.edu.pk
178LogLevel warn
--- /usr/local/apache/conf/httpd.conf.work.O60GBbOFSwotmcqu ---


Rebuilding configuration without any local modifications.

Failed to generate a syntactically correct Apache configuration.
Bad configuration file located at /usr/local/apache/conf/httpd.conf.work.O60GBbOFSwotmcqu
Error:
[Tue Nov 19 12:59:15 2013] [crit] (EAI 9)Address family for hostname not supported: alloc_listener: failed to set up sockaddr for ::

Configuration problem detected on line 172 of file /usr/local/apache/conf/httpd.conf.work.O60GBbOFSwotmcqu: Listen setup failed

--- /usr/local/apache/conf/httpd.conf.work.O60GBbOFSwotmcqu ---
166# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
167
168
169PidFile logs/httpd.pid
170LockFile logs/accept.lock
171Listen 0.0.0.0:80
172 ===> Listen [::]:80 <===
173User nobody
174Group nobody
175ExtendedStatus On
176ServerAdmin webmaster@kmu.edu.pk
177ServerName server.kmu.edu.pk
178LogLevel warn
--- /usr/local/apache/conf/httpd.conf.work.O60GBbOFSwotmcqu ---



Restarting httpd (hard)...

Apache successfully restarted.

Done!

As you seem to show only trouble with WHM functionality I'd say best check out that products own forum.

You just need to run EasyApche to fix it.

From command line:

/scripts/easyapache

or from WHM, you can select easy apache menu to fix it.