LinuxQuestions.org
Welcome to the most active Linux Forum on the web.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - General
User Name
Password
Linux - General This Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.

Notices

Reply
 
Search this Thread
Old 01-22-2007, 06:52 PM   #1
farmerjoe
Member
 
Registered: Oct 2004
Location: Texas
Distribution: Ubuntu - Home, RHEL4 - Server
Posts: 96

Rep: Reputation: 15
System Suddenly Overloading...please help!


I have a webserver and just this morning it seems to have started to overload.

As of this morning, my server has been running at an abnormally high load due to what seems to be a whole lot of httpd connections. However, we are receiving no more traffic to our websites than normal, so I am sure it is not related to our website traffic. Also I have not changed any settings nor have I edited any of our website scripts. So none of these things could be causing this. We normally run at under 1% server load. When I examine our apache access-log file, i see that there are thousands of ""GET /whm-server-status HTTP/1.0" 200" messages. Could this be the culprit?

Also, the /var/log/secure file has thousands of lines of this:
Jan 21 09:30:12 ns3 xinetd[2374]: START: imap pid=4648 from=127.0.0.1
Jan 21 09:38:35 ns3 xinetd[2374]: START: imap pid=6216 from=127.0.0.1
Jan 21 09:46:58 ns3 xinetd[2374]: START: imap pid=9846 from=127.0.0.1
Jan 21 09:55:21 ns3 xinetd[2374]: START: imap pid=11350 from=127.0.0.1

Here is an export from a TOP i just ran:
18:50:56 up 56 min, 2 users, load average: 127.47, 106.16, 66.71
913 processes: 910 sleeping, 2 running, 1 zombie, 0 stopped
CPU states: cpu user nice system irq softirq iowait idle
total 13.1% 0.0% 10.4% 0.0% 0.4% 74.2% 1.6%
cpu00 16.9% 0.0% 6.0% 0.0% 0.8% 76.0% 0.0%
cpu01 13.0% 0.0% 10.1% 0.0% 0.0% 72.0% 4.8%
cpu02 10.7% 0.0% 8.9% 0.1% 0.7% 79.4% 0.0%
cpu03 11.9% 0.0% 16.7% 0.0% 0.0% 69.6% 1.6%
Mem: 2074544k av, 1982340k used, 92204k free, 0k shrd, 25936k buff
1612984k active, 226180k inactive
Swap: 2097136k av, 878380k used, 1218756k free 332448k cached

PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU COMMAND
17569 root 22 0 5888 1872 4888 R 2.6 0.0 0:04 3 top
3568 root 0 -20 0 0 0 DW< 0.4 0.0 1:54 2 loop0
17060 root 15 0 0 0 0 SW 0.4 0.0 0:00 2 pdflush
6877 nobody 16 0 17448 8280 15068 D 0.2 0.3 0:00 0 httpd
6923 nobody 16 0 17464 8184 15068 D 0.2 0.3 0:01 0 httpd
7728 nobody 16 0 17852 8644 15068 D 0.2 0.4 0:02 0 httpd
9116 nobody 16 0 17448 8252 15068 D 0.2 0.3 0:00 0 httpd
17560 root 19 0 7452 3872 6180 S 0.2 0.1 0:00 2 exim
17572 root 20 0 7436 3880 6180 S 0.2 0.1 0:00 0 exim
2966 nobody 16 0 17504 8316 15068 S 0.1 0.4 0:01 3 httpd
6492 nobody 16 0 18148 8784 15068 S 0.1 0.4 0:01 2 httpd
6593 nobody 15 0 17448 8284 15068 R 0.1 0.3 0:03 3 httpd
6601 nobody 16 0 18148 8860 15068 D 0.1 0.4 0:00 3 httpd
6687 nobody 15 0 17448 8252 15068 S 0.1 0.3 0:01 0 httpd
6935 nobody 15 0 17472 8260 15068 S 0.1 0.3 0:02 0 httpd
7192 nobody 16 0 18240 8828 15068 D 0.1 0.4 0:03 0 httpd
7197 nobody 16 0 18068 8840 15068 D 0.1 0.4 0:03 0 httpd
7223 nobody 16 0 17472 8308 15068 D 0.1 0.4 0:02 0 httpd
7236 nobody 16 0 17908 8656 15068 D 0.1 0.4 0:01 3 httpd
7481 nobody 16 0 18040 8768 15068 D 0.1 0.4 0:01 1 httpd
7549 nobody 15 0 18852 8808 15068 D 0.1 0.4 0:01 0 httpd
7631 nobody 17 0 17792 8600 15068 D 0.1 0.4 0:01 0 httpd
7636 nobody 17 0 18052 8808 15068 D 0.1 0.4 0:02 0 httpd
7701 nobody 16 0 17980 8756 15068 D 0.1 0.4 0:04 0 httpd
7726 nobody 16 0 17856 8628 15068 S 0.1 0.4 0:02 1 httpd
9880 nobody 16 0 17956 8696 15068 D 0.1 0.4 0:02 1 httpd
17586 root 25 0 7568 3908 6180 S 0.1 0.1 0:00 1 exim
1 root 16 0 1508 456 1356 S 0.0 0.0 0:12 3 init
2 root RT 0 0 0 0 SW 0.0 0.0 0:00 0 migration/0
3 root 34 19 0 0 0 SWN 0.0 0.0 0:00 0 ksoftirqd/0
4 root RT 0 0 0 0 SW 0.0 0.0 0:00 1 migration/1
5 root 34 19 0 0 0 SWN 0.0 0.0 0:00 1 ksoftirqd/1
6 root RT 0 0 0 0 SW 0.0 0.0 0:00 2 migration/2


I am stumped and don't know what else to do. I've rebooted a couple of times. Restarted apache server...mysqld server...exim server....etc. I really need to get this fixed as there are a few high traffic websites on this server i need to have up. Pleasssseeee help guys!


-farmerjoe
 
Old 01-22-2007, 07:43 PM   #2
unSpawn
Moderator
 
Registered: May 2001
Posts: 27,367
Blog Entries: 54

Rep: Reputation: 2866Reputation: 2866Reputation: 2866Reputation: 2866Reputation: 2866Reputation: 2866Reputation: 2866Reputation: 2866Reputation: 2866Reputation: 2866Reputation: 2866
Do you run anything that should check /whm-server-status?
Do the /whm-server-status queries come from one or multiple IP addresses?
What happens if you block access to /whm-server-status?
Is the box swamped in outbound traffic? To remote port TCP/25 by any chance?
Did you run a Chkrootkit / Rootkit Hunter check on the system (just in case)?
Otherwise checked for "weird" stuff in the system and daemon logs?
Otherwise checked for "weird" stuff in /tmp, /var/tmp and any other temp dirs apps are allowed to write?
 
Old 01-22-2007, 07:52 PM   #3
Jerre Cope
Member
 
Registered: Oct 2003
Location: Texas (central)
Distribution: ubuntu,Slackware,knoppix
Posts: 323

Rep: Reputation: 37
What does our old friend netstat show?

I'm not an an Internet Service Provider guru, but I think I would:

1) Limit all services, such as IMAP, smtp, etc to the WAN, either by disabling or firewall.
2) Take several netstat snap shops, sort and diff them to see if you can see a incoming source.
3) Maybe use ethereal for a similar purpose.
4) Try the rate limiting features in the Linux firewall.
 
Old 01-23-2007, 09:36 AM   #4
farmerjoe
Member
 
Registered: Oct 2004
Location: Texas
Distribution: Ubuntu - Home, RHEL4 - Server
Posts: 96

Original Poster
Rep: Reputation: 15
the whm check is a local check. Its something Cpanel does to make sure its still working.

I think I got this problem solved....or at least temporarily solved. Most of the HTTPD requests seemed to be coming from SSL connections. I started apache without SSL and that seemed to immediately solve the load problem. We have no need for SSL, so this is a temporary solution at least.

Thanks so much for your tips. I'll update this thread if the problems resume today.
 
  


Reply

Tags
analysis, help, log, troubleshooting


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
system time suddenly changed selva_pondy Linux - Hardware 7 10-28-2007 11:44 AM
system clock suddenly slow each night jptxs Linux - Software 12 01-06-2007 07:46 PM
Suddenly my system takes twice the memory Imerion Linux - General 5 12-21-2005 06:19 PM
all file system went suddenly Read Only hungry_linux Linux - Security 2 07-02-2005 11:11 AM
System suddenly shutting down and installing modules blandman Linux - Laptop and Netbook 0 08-05-2004 08:20 PM


All times are GMT -5. The time now is 05:59 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration