hi,
my RH AS 3 (Update 8 with all patches) server hung when i was trying to invoke "ps".

the server runs more than 25000 processes under medium load.

kernel: 2.4.21-47.0.1.ELhugemem SMP
mem: 16GB
cpu: AMD Opteron(tm) Processor 844 x 4

any suggestions? thank you very much!

Ummm - don't do it ???.
What does "hung up" mean ??? - CPU maxed out, disks hammered, swap thrashing, all/none of the above ...

What option(s) were you using on the command ?.
How about "top", does it do likewise ?.
What about "ps -r"

Hi, thanks for your reply.

I use "ps aux" and then the system did not respond. I can't find what happened because i cannot log in from neither ssh nor tty.

I've no idea if top would do likewise.

Thanks!

Mmmmm - not good.
With RH, you should have sysstat I would imagine. With the sar (and sadc) componentyou can get a history of what happens on your system.
Perhaps have a look at that.

Hi, i just check sar's report but found nothing abnormal.
Is it something related to some kernel parameters or its limit/defect when inquiring too many processes under /proc?

btw: a quick question. what's the difference between /proc/pid and /proc/.pid ?

thanks!

The latter I would imagine.
As for the other query, I have no idea - I just see /proc/<pid>

Can you paste me the output of the following command on your server?

/usr/bin/strace ps -ef

This will give you an idea on the point where the ps command hangs up. The problem is due to some defunct threads running on the server. defunct threads are the one which reside in the system even after the parent processes are killed. I believe it could be due to some kernel module not working correctly.

What does this server is used for?
Your ps command hangs up since it tries to read some processes from /proc directory, which are zombies.

Rahul Khare.

Hi, Thanks for your reply. This is really a good idea.
Anyway, it's unpredictable when the box hangs. I'm so unlucky that it never hang when i'm there...
In fact, i've tried to trace syscall ps uses.

the server is used for a critical business use, which runs lots of processes (also lots of threads)

and... I'd like to ask again about /proc/.pid what's the difference with common /proc/pid?

thank you all!

When you say the box hangs, does the OS actually crash and kernel panic or does it just lock/freeze? If you have another RH ox on the same LAN it would be worth setting up netdump to see if that will provide you with a useful memory and syslog dump of the failure. Have you setup netdump in the past? - it's *really* easy but fantastic for this kind of problem. Also, if you need to raise a call with RedHat then you can provide them all the details they need to sort the problem out for you, assuming you have support that is.

Other thoughts - has anything changed recently no matter how small or 'unrealted'? Updated applications, new network switches, etc etc?

As for /proc/pid vs /proc/.pid - I've only got /proc/pid on my 2.4 and 2.6 boxes which is odd.

HTH

Jamie

Hi, i've set up netdump. Hopefully it could help!

Thanks anyway.

i think i've got some ideas about /proc/.pid

they are "threads", which belong to a specific process id

It seems that "ps" stopped when reading a particular process' process status file (/proc/pid/stat)

...