top output

grob115 · 12-19-2010, 09:58 AM

Thanks. This is unlikely due to the following two reasons:
1) There is a very good hardware firewall in front of the server.
2) Each time I restart Apache, things go back to normal. I'd imagine the DDOS source to persist and Apache after restarts would still be busy.
3) The process that is busy, is not the process that forks out the worker processes. It's one of the work processes that's busy.

httpd.conf has the following:
LogLevel warn

Any idea if I need to restart when I change the LogLevel in order to make it effective? If so, that means I wouldn't be able to turn on additional logging without restarting, which stops the symptoms.

There's no official documentation on how to upgrade, is there? I'm using 2.2.15 and they have 2.2.17 now.

stress_junkie · 12-19-2010, 05:25 PM

You could still watch the network packets.

grob115 · 01-04-2011, 10:36 AM

Can't believe this is happening. In fact, this time it must have been on going for a while. Have generated a dump file up to about 1.2MB. Not sure how to read it though. Any advise?

Code:

[root@production ~]# tcpdump -w /root/tcpdump_output04Jan11_08:30PST.txt
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
15072 packets captured
15116 packets received by filter
44 packets dropped by kernel

Look at the ridiculous load average and process time.....

Code:

top - 08:25:20 up 146 days, 11:19,  1 user,  load average: 7.16, 7.34, 7.39
Tasks: 141 total,   9 running, 132 sleeping,   0 stopped,   0 zombie
Cpu0  : 74.3%us, 24.3%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.7%hi,  0.7%si,  0.0%st
Cpu1  : 69.7%us, 30.3%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   2059516k total,  1886644k used,   172872k free,   159632k buffers
Swap:  4095992k total,       84k used,  4095908k free,   779156k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
19443 daemon    25   0  115m  18m 2876 R 30.0  0.9   1408:50 httpd
13874 daemon    25   0  115m  18m 2908 R 27.6  0.9   8199:18 httpd
11156 daemon    25   0  115m  18m 3052 R 26.6  0.9   1631:02 httpd
13868 daemon    25   0  115m  18m 2984 R 26.6  0.9   8341:24 httpd
19199 daemon    25   0  115m  17m 2776 R 26.3  0.9   2407:59 httpd
14921 daemon    25   0  115m  18m 2988 R 24.0  0.9   1613:19 httpd
13405 daemon    25   0  116m  18m 3116 R 23.6  0.9   8305:59 httpd
11829 daemon    15   0  107m 9.8m 2836 S  5.0  0.5   0:00.15 httpd
11830 daemon    16   0  105m 8420 2796 S  3.3  0.4   0:00.10 httpd
11831 daemon    16   0  107m 9960 2796 S  3.3  0.5   0:00.10 httpd
11748 daemon    16   0  116m  18m 3016 S  1.3  0.9   0:00.64 httpd
11780 daemon    15   0  107m 9.8m 2836 S  0.7  0.5   0:00.09 httpd
11786 daemon    15   0  107m 9.8m 2840 S  0.7  0.5   0:00.12 httpd
11804 daemon    15   0  107m 9.8m 2872 S  0.7  0.5   0:00.11 httpd
11608 daemon    16   0  110m  13m 3004 S  0.3  0.7   0:01.06 httpd
11813 daemon    15   0  106m 8940 2588 S  0.3  0.4   0:00.01 httpd

unixfool · 01-25-2011, 01:19 PM

Quote:

Originally Posted by grob115

Can't believe this is happening. In fact, this time it must have been on going for a while. Have generated a dump file up to about 1.2MB. Not sure how to read it though. Any advise?

Code:

[root@production ~]# tcpdump -w /root/tcpdump_output04Jan11_08:30PST.txt
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
15072 packets captured
15116 packets received by filter
44 packets dropped by kernel

Use the following, as it should enable you to read it:

tcpdump -r /root/tcpdump_output04Jan11_08:30PST.txt

-w = write
-r = read

Note that although you captured 1.2MB, you didn't filter anything from your dump...you captured everything, including non-Apache traffic. This is good, in case it is something that's affecting more than Apache.