| jvanv8 |
03-05-2011 04:57 PM |
5-10 Nobody httpd processes consume CPU, load
I have 2 load balanced servers that are both experiencing high load due to "nobody" httpd processes that cause load to ramp up to 80-100+ until httpd is restarted (and then it returns)
Code:
top - 16:42:13 up 3:15, 1 user, load average: 40.41, 41.19, 39.28
Tasks: 366 total, 38 running, 321 sleeping, 6 stopped, 1 zombie
Cpu(s): 18.7%us, 81.2%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.1%si, 0.0%st
Mem: 8174168k total, 3693952k used, 4480216k free, 202440k buffers
Swap: 2104504k total, 0k used, 2104504k free, 2506036k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
20194 nobody 15 0 280m 110m 6064 R 51.1 1.4 5:04.97 httpd
23152 nobody 25 0 185m 15m 4264 R 26.9 0.2 60:30.39 httpd
15156 nobody 25 0 186m 16m 4932 R 26.6 0.2 6:10.44 httpd
7845 nobody 25 0 184m 14m 4824 R 26.2 0.2 12:05.19 httpd
27697 nobody 25 0 184m 13m 4592 R 25.6 0.2 38:52.67 httpd
17269 nobody 25 0 187m 17m 4776 R 24.9 0.2 4:31.01 httpd
27560 nobody 25 0 184m 13m 4780 R 24.6 0.2 38:00.86 httpd
3312 nobody 25 0 182m 10m 4100 R 24.2 0.1 18:02.83 httpd
14440 nobody 25 0 182m 10m 4072 R 23.9 0.1 7:09.11 httpd
16708 nobody 25 0 183m 13m 4764 R 23.6 0.2 5:29.71 httpd
30853 nobody 25 0 187m 16m 4760 R 23.6 0.2 30:21.56 httpd
11045 nobody 25 0 182m 11m 4836 R 23.2 0.1 8:15.32 httpd
15288 nobody 25 0 185m 15m 4824 R 23.2 0.2 6:15.49 httpd
18056 nobody 25 0 187m 16m 4904 R 23.2 0.2 3:31.00 httpd
22790 nobody 25 0 182m 10m 4012 R 23.2 0.1 62:16.74 httpd
1634 nobody 25 0 182m 10m 4044 R 22.9 0.1 22:22.69 httpd
14460 nobody 25 0 182m 10m 3924 S 22.9 0.1 7:19.62 httpd
21521 nobody 25 0 184m 13m 4896 R 21.6 0.2 61:31.62 httpd
28403 nobody 25 0 184m 13m 4768 R 20.6 0.2 35:56.49 httpd
8092 nobody 25 0 182m 10m 4092 R 20.3 0.1 12:20.71 httpd
17511 nobody 25 0 186m 16m 4836 R 20.3 0.2 4:11.25 httpd
21051 nobody 25 0 186m 16m 4784 R 20.3 0.2 66:57.50 httpd
28520 nobody 25 0 184m 13m 4760 R 20.3 0.2 35:35.02 httpd
1997 nobody 25 0 184m 13m 4528 R 19.9 0.2 20:59.58 httpd
8266 nobody 25 0 182m 11m 4408 R 19.9 0.1 12:07.31 httpd
11429 nobody 25 0 185m 15m 4612 R 19.9 0.2 8:35.06 httpd
12654 nobody 25 0 184m 13m 4720 R 19.9 0.2 8:03.82 httpd
27974 nobody 25 0 182m 11m 4428 S 19.9 0.1 37:52.26 httpd
31684 nobody 25 0 182m 11m 4248 R 19.9 0.1 27:36.75 httpd
31749 nobody 25 0 182m 10m 4000 R 19.9 0.1 28:01.86 httpd
3378 nobody 25 0 185m 15m 4832 R 19.6 0.2 17:41.87 httpd
22065 mysql 17 0 290m 51m 4116 S 18.9 0.6 0:00.80 mysqld
19951 nobody 15 0 307m 135m 6244 R 15.6 1.7 3:02.78 httpd
21908 nobody 15 0 182m 12m 4780 S 9.6 0.2 0:02.25 httpd
8455 nobody 25 0 182m 10m 4064 R 5.0 0.1 11:41.38 httpd
22010 nobody 15 0 185m 15m 4556 S 4.3 0.2 0:00.21 httpd
27443 nobody 25 0 184m 13m 4884 R 3.7 0.2 36:55.42 httpd
22068 mysql 16 0 290m 51m 4116 S 3.3 0.6 0:00.10 mysqld
27958 nobody 25 0 182m 11m 4768 R 3.3 0.1 37:44.92 httpd
21777 nobody 15 0 184m 14m 4832 S 3.0 0.2 0:00.67 httpd
21927 nobody 15 0 185m 14m 3844 S 3.0 0.2 0:00.26 httpd
22018 mysql 16 0 290m 51m 4116 S 2.3 0.6 0:00.19 mysqld
21941 nobody 15 0 185m 15m 4824 S 1.7 0.2 0:00.84 httpd
21789 nobody 15 0 202m 31m 4884 S 1.3 0.4 0:03.49 httpd
21817 nobody 15 0 184m 14m 4832 S 1.3 0.2 0:00.77 httpd
21916 nobody 15 0 0 0 0 Z 0.3 0.0 0:00.41 httpd <defunct>
.... [truncated] ...
If I do an strace on any of these processes I get:
Code:
poll([{fd=91, events=POLLIN|POLLPRI}], 1, 0) = 0 (Timeout)
clock_gettime(CLOCK_MONOTONIC, {3140, 207436246}) = 0
....{with similar commands thousands and thousands of times in each thread}....
Any idea what could cause this? |
