Linux stopped processes

eyanu · 11-12-2012, 02:38 AM

Hullo, i would really need your help, of recent my server load has been really high:

Quote:

top - 09:35:47 up 25 days, 17:46, 2 users, load average: 12.38, 14.71, 19.82
Tasks: 747 total, 4 running, 596 sleeping, 141 stopped, 6 zombie
Cpu(s): 24.9%us, 36.7%sy, 28.0%ni, 7.6%id, 2.3%wa, 0.1%hi, 0.3%si, 0.0%st

and as you can see there are 141 stopped processes, which i think are the cause, though i execute

Quote:

jobs

it returns nothing, how do i kill the stopped processes or how do i list them? Any help is greatly appreciated.

chrism01 · 11-12-2012, 06:09 AM

Show the rest of the top output ie include the process table.
You may find this useful http://slack-linux.blogspot.com.au/2...ate-codes.html
You can use it to figure out what is going on; don't just start killing procs, you'll damage your system/data.

tronayne · 11-12-2012, 06:36 AM

Well, this would be a good time for a reboot... and a good time to take a hard look at the processes you're staring up (if they're "home-brew" anyway). For example, top looks like this with only me doing anything right now:

Code:

top - 06:46:20 up 17 days, 18:28,  2 users,  load average: 0.00, 0.01, 0.05
Tasks: 158 total,   1 running, 156 sleeping,   0 stopped,   1 zombie
Cpu(s):  1.0%us,  0.7%sy,  0.0%ni, 98.3%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   7994668k total,  4228468k used,  3766200k free,   235296k buffers
Swap: 15623208k total,    37680k used, 15585528k free,  3142668k cached

(and I need to go find that zomibe and get it dealt with ... OK found it, it's hp-update, deal with it later).

The top column you're interested in is the one labeled S (Process Status); the status can be one of

'D' = uninterruptible sleep
'R' = running
'S' = sleeping
'T' = traced or stopped
'Z' = zombie

Note that the first column in a top display is PID; you can use that PID with ps -ef | grep PID (where "PID" is the PID number).

There's a number of ways you can get zombies (a child process that hasn't died when the parent exits for one). Google for "linux zombie process" for a lot of reference material. You can use the ps and grep utilities to find them; e.g.,

Code:

ps -ef | grep defunct
trona    30977 25333  0 06:51 ?        00:00:01 [python] <defunct>

shows me the zombie sitting on my system (you're going to see a lot more).

That's why the "home built" remark earlier -- if you've got software that uses "fork and exec" to launch a child process you need to take a look at making sure that when the parent dies that the child dies too. There are processes where the parent on purpose launches a child then dies without killing the child (that's what daemons do) and you need to look at those and make sure that there is some sort of error trap in the child that will cause it to die in a daemon.

On the other hand, if you're running third party applications, you may need to look into those as well (you should not have zombies at all and you should not have anywhere near that number of stopped processes). Those stopped processes may be waiting for some unavailable resource and that will take some looking.

Might be a good idea to reboot (which will clear out all that stuff) and review what daemons you're starting, what applications are starting, what users are doing (like leaving an application running when they leave for the day and not logging out). You know, reboot it and keep an eye on things, can't hurt, might help.

Hope this helps some.

unSpawn · 11-12-2012, 07:29 AM

Quote:

Originally Posted by tronayne

You can use the ps and grep utilities to find them;

...but use the right ps output modifiers instead:

Code:

/bin/ps ax -eos,state,pid,ppid,args|egrep "^(Z|X|T)"

eyanu · 11-12-2012, 09:11 AM

ps -ef | grep defunct

Quote:

travacom 17453 24222 0 Nov08 ? 00:00:00 [sh] <defunct>
travacom 17454 21945 0 Nov08 ? 00:00:00 [sh] <defunct>
travacom 17457 29514 0 Nov08 ? 00:00:00 [sh] <defunct>
travacom 17467 28995 0 Nov08 ? 00:00:00 [sh] <defunct>
root 29335 16091 0 16:08 pts/2 00:00:00 grep defunct

ps ax -eos,state,pid,ppid,args|egrep "^(Z|X|T)"

Quote:

T T 392 1 /usr/bin/php-cgi -c /var/www/vhosts/iucea.org/etc/php.ini
T T 494 1 /usr/sbin/
T T 498 1 /usr/local/apache/bin/httpd -DSSL

? ssstt
T T 620 1 /usr/bin/php-cgi -c /var/www/vhosts/iucea.org/etc/php.ini
T T 952 1 /usr/bin/php-cgi -c /var/www/vhosts/iucea.org/etc/php.ini
T T 1165 1 /usr/bin/php-cgi -c /var/www/vhosts/iucea.org/etc/php.ini
T T 1806 1 /usr/bin/php-cgi -c /var/www/vhosts/iucea.org/etc/php.ini
T T 2032 1 /usr/bin/php-cgi -c /var/www/vhosts/iucea.org/etc/php.ini
T T 2370 1 /usr/bin/php-cgi -c /var/www/vhosts/iucea.org/etc/php.ini
T T 2582 1 /usr/bin/php-cgi -c /var/www/vhosts/iucea.org/etc/php.ini
T T 2799 1 /usr/bin/php-cgi -c /var/www/vhosts/iucea.org/etc/php.ini
T T 2801 1 /usr/bin/php-cgi -c /var/www/vhosts/iucea.org/etc/php.ini
T T 2911 1 /usr/bin/php-cgi -c /var/www/vhosts/iucea.org/etc/php.ini
T T 3027 1 /usr/bin/php-cgi -c /var/www/vhosts/iucea.org/etc/php.ini
T T 3028 1 /usr/bin/php-cgi -c /var/www/vhosts/iucea.org/etc/php.ini
T T 3138 1 /usr/bin/php-cgi -c /var/www/vhosts/iucea.org/etc/php.ini
T T 3253 1 /usr/bin/php-cgi -c /var/www/vhosts/iucea.org/etc/php.ini
T T 3360 1 /usr/bin/php-cgi -c /var/www/vhosts/iucea.org/etc/php.ini
T T 3473 1 /usr/bin/php-cgi -c /var/www/vhosts/iucea.org/etc/php.ini
T T 3699 1 /usr/bin/php-cgi -c /var/www/vhosts/iucea.org/etc/php.ini
T T 3749 1 /usr/bin/php-cgi -c /var/www/vhosts/iucea.org/etc/php.ini
T T 4030 1 /usr/bin/php-cgi -c /var/www/vhosts/iucea.org/etc/php.ini
T T 4139 1 /usr/bin/php-cgi -c /var/www/vhosts/iucea.org/etc/php.ini
T T 4471 1 /usr/bin/php-cgi -c /var/www/vhosts/iucea.org/etc/php.ini
T T 4708 1 /usr/bin/php-cgi -c /var/www/vhosts/iucea.org/etc/php.ini
T T 4709 1 /usr/bin/php-cgi -c /var/www/vhosts/iucea.org/etc/php.ini
T T 4926 1 /usr/bin/php-cgi -c /var/www/vhosts/iucea.org/etc/php.ini
T T 5098 1 /usr/bin/php-cgi -c /var/www/vhosts/iucea.org/etc/php.ini
T T 5142 1 /usr/bin/php-cgi -c /var/www/vhosts/iucea.org/etc/php.ini
T T 5143 1 /usr/bin/php-cgi -c /var/www/vhosts/iucea.org/etc/php.ini
T T 5145 1 /usr/bin/php-cgi -c /var/www/vhosts/iucea.org/etc/php.ini
T T 5146 1 /usr/bin/php-cgi -c /var/www/vhosts/iucea.org/etc/php.ini
T T 5147 1 /usr/bin/php-cgi -c /var/www/vhosts/iucea.org/etc/php.ini
T T 5148 1 /usr/bin/php-cgi -c /var/www/vhosts/iucea.org/etc/php.ini
T T 5366 1 /usr/bin/php-cgi -c /var/www/vhosts/iucea.org/etc/php.ini

eyanu · 11-12-2012, 09:16 AM

Ok i've killed processes under

Quote:

ps -ef | grep defunct

and they have dropped to 137 processes, how do i kill processes under:

Quote:

ps ax -eos,state,pid,ppid,args|egrep "^(Z|X|T)"

eyanu · 11-12-2012, 09:26 AM

Thanks guys, i finally cleared the zombies, wouldn't have done with out you.

Quote:

top - 16:25:29 up 26 days, 36 min, 2 users, load average: 6.49, 13.40, 19.10
Tasks: 567 total, 2 running, 565 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.5%us, 5.6%sy, 15.5%ni, 68.6%id, 9.5%wa, 0.1%hi, 0.2%si, 0.0%st
Mem: 32814456k total, 7898232k used, 24916224k free, 535300k buffers
Swap: 13631376k total, 680632k used, 12950744k free, 1505052k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
15867 travacom 24 4 737m 90m 44m S 16.2 0.3 0:00.49 /usr/bin/php-cgi -c /var/www/vhosts/traveltoeastafrica.com/etc/php
29020 mysql 26 6 5039m 1.1g 7940 S 10.2 3.5 8922:33 /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --plug
10686 root 20 0 1483m 481m 660 S 1.7 1.5 1900:19 /usr/sbin/sw-collectd -C /etc/sw-collectd/collectd.conf
32540 apache 24 4 1474m 495m 37m S 1.3 1.5 0:05.28 /usr/sbin/httpd
7075 apache 24 4 1279m 324m 60m S 1.0 1.0 0:06.18 /usr/sbin/httpd
15984 root 30 10 32524 5200 1992 R 1.0 0.0 0:00.03 /usr/bin/perl /var/www/cgi-bin/awstats/awstats.pl -config=heoso.or
28847 apache 24 4 1474m 514m 55m S 1.0 1.6 0:22.66 /usr/sbin/httpd
8915 root 20 0 15424 1724 996 R 0.7 0.0 0:00.48 top
23686 apache 24 4 1281m 298m 35m S 0.7 0.9 0:02.69 /usr/sbin/httpd
28407 apache 24 4 1375m 347m 62m S 0.7 1.1 0:11.11 /usr/sbin/httpd
28531 apache 24 4 1279m 318m 54m S 0.7 1.0 0:07.20 /usr/sbin/httpd
30 root RT 0 0 0 0 S 0.3 0.0 0:01.66 [watchdog/6]
385 root 20 0 0 0 0 S 0.3 0.0 334:50.37 [md1_raid1]
2687 root 20 0 208m 32m 1872 S 0.3 0.1 2:06.95 lfd - sleeping
3609 root 20 0 105m 1056 684 S 0.3 0.0 4:56.68 bash
4088 root 20 0 105m 748 548 S 0.3 0.0 10:04.73 bash
4565 root 20 0 105m 752 548 S 0.3 0.0 8:23.84 bash
12285 root 20 0 105m 856 652 S 0.3 0.0 6:38.20 bash
15952 root 30 10 26648 3492 1940 S 0.3 0.0 0:00.01 /usr/bin/perl /usr/share/awstats/awstats_buildstaticpages.pl -awst
28840 apache 24 4 1279m 305m 41m S 0.3 1.0 0:23.00 /usr/sbin/httpd
29284 apache 24 4 1220m 262m 62m S 0.3 0.8 0:08.53 /usr/sbin/httpd
1 root 20 0 19328 680 448 S 0.0 0.0 0:09.69 /sbin/init
2 root 20 0 0 0 0 S 0.0 0.0 0:00.02 [kthreadd]
3 root RT 0 0 0 0 S 0.0 0.0 0:50.98 [migration/0]
4 root 20 0 0 0 0 S 0.0 0.0 0:20.65 [ksoftirqd/0]
5 root RT 0 0 0 0 S 0.0 0.0 0:00.00 [migration/0]