[SOLVED] Too many forks, pid got used, we're confused

Aquarius_Girl · 08-31-2010, 08:15 AM

OpenSuse 11.2
LXDE

A very strange thing happened just now.

I was in the GUI mode.
The processes started by me on my computer were:
1. OpenOffice word.
2. Firefox.
3. Konsole.
4. A Pdf and a Folder.

I started feeling slowly that the system was getting slow in responding.

I did:

Code:

anisha@linux-uitj:~> ps -el
...
...
0 S  1000   300 32767  0  80   0 -  3106 pipe_w ?        00:00:00 sh            
0 S  1000   301 32767  0  80   0 -  1228 pipe_w ?        00:00:00 sed           
1 S  1000   304   300  0  80   0 -  3106 wait   ?        00:00:00 sh            
0 S  1000   305   304  0  80   0 -  3106 pipe_w ?        00:00:00 sh            
0 S  1000   307   304  0  80   0 -  1228 pipe_w ?        00:00:00 sed           
1 S  1000   309   305  0  80   0 -  3106 wait   ?        00:00:00 sh            
0 S  1000   310   309  0  80   0 -  3106 pipe_w ?        00:00:00 sh                                     
0 S  1000   312   309  0  80   0 -  1228 pipe_w ?        00:00:00 sed                                    
1 S  1000   314   310  0  80   0 -  3106 wait   ?        00:00:00 sh                                     
0 S  1000   315   314  0  80   0 -  3106 pipe_w ?        00:00:00 sh                                     
1 S  1000   318   315  0  80   0 -  3106 wait   ?        00:00:00 sh                                     
0 S  1000   319   314  0  80   0 -  1228 pipe_w ?        00:00:00 sed                                    
0 S  1000   320   318  0  80   0 -  3106 pipe_w ?        00:00:00 sh                                     
0 S  1000   322   318  0  80   0 -  1228 pipe_w ?        00:00:00 sed                                    
1 S  1000   324   320  0  80   0 -  3106 wait   ?        00:00:00 sh                                     
0 S  1000   325   324  0  80   0 -  3106 pipe_w ?        00:00:00 sh

I have presented a short list here, actual list was very long !!!!

I pressed Ctrl+Alt+Backspace to get out !!!!

It too 3 minutes and I noticed a message on the non-GUI mode :

Quote:

Too many forks, pid got used, we're confused

What must be the cause, any ideas ???????????

druuna · 08-31-2010, 08:40 AM

Hi,

Looking at the mother and child pids in the output you posted it looks like and endless loop was started that ate all the available pid numbers the system has.

WARNING! The following example may crash your computer if executed.

:(){ :|:& };:

The above is a known Bash fork() bomb that can produce something similar.

Most of the time it is very hard to do anything useful after something like this happens (you won't be able to get any commands to execute).

Not much to go on, but maybe it clarifies the behaviour a bit.

unSpawn · 08-31-2010, 09:02 AM

BTW, posting something like '/bin/ps U 1000 xfo pid,ppid,uid,cmd --sort=ppid' (or 'pstree -Apuh `id -gn`') would provide more process info making things easier to read.

Aquarius_Girl · 08-31-2010, 09:17 AM

Quote:

Originally Posted by unSpawn

BTW, posting something like '/bin/ps U 1000 xfo pid,ppid,uid,cmd --sort=ppid' (or 'pstree -Apuh `id -gn`') would provide more process info making things easier to read.

Thanks for replying !

But during that incident I could barely manage to save this output on a text file, I couldn't do anything more as mouse clicks were not responding !!

I had to hard restart the computer, so I couldn't execute pstree !

druuna · 08-31-2010, 09:21 AM

Quote:

Originally Posted by anishakaul

But during that incident I could barely manage to save this output on a text file, I couldn't do anything more as mouse clicks were not responding !!

I had to hard restart the computer, so I couldn't execute pstree !

That is a classic symptom of what I described in my previous post, be glad you could at least do that command and save the output

Aquarius_Girl · 08-31-2010, 09:28 AM

Quote:

Originally Posted by druuna

WARNING! The following example may crash your computer if executed.

:(){ :|:& };:

The above is a known Bash fork() bomb that can produce something similar.

Thanks for replying !

I was confused and worried because I had not manually typed any such command !!

I should have checked history command !!

but how could I ???

druuna · 08-31-2010, 09:39 AM

Hi,

Unless you remove your history during shut-down/boot it is still there the next time you boot up your machine......

The bash fork bomb I mentioned is just an example (it is fast! and you can probably do nothing of use any more within seconds if not faster) and I'm pretty sure that specific example wasn't the cause.

Aquarius_Girl · 08-31-2010, 09:44 AM

Quote:

Originally Posted by druuna

Unless you remove your history during shut-down/boot it is still there the next time you boot up your machine......

Yes, how could I forget that

Perhaps because of the goose bumps I am still having !

Aquarius_Girl · 08-31-2010, 09:59 AM

Quote:

Originally Posted by druuna

:(){ :|:& };:

The above is a known Bash fork() bomb that can produce something similar.

Right now I am at home not in the office, out of curiosity, I just tried the above command thinking I stop it with ctrl+c

But the command was over-smart it ran in the background and didn't give me even a chance to type the kill command

druuna · 08-31-2010, 10:07 AM

Hi,

Have a look here Understanding Bash fork() bomb

You're not up against 1 process. 1 becomes 2 -> 4 -> 8 -> 16 etc

unSpawn · 09-01-2010, 03:21 AM

Quote:

Originally Posted by anishakaul

during that incident I could barely manage to save this output on a text file, I couldn't do anything more as mouse clicks were not responding !!
I had to hard restart the computer, so I couldn't execute pstree !

Understandable, yes. I'm only saying that with the right information it would be easier to find out what on your system caused it. Right now the only thing one can say is that applying ulimits is always good. You did apply those, right?

Aquarius_Girl · 09-01-2010, 05:04 AM

Quote:

Originally Posted by druuna

I'm pretty sure that specific example wasn't the cause.

I checked the history command, there was nothing suspicious.

Then I suddenly remembered that there was one more process opened by me and that was KDevelop ! I ran commands to compile our project there. There is a file named "Libtool" which needs to be modified each time ./configure runs other wise the system hangs !!

That time I must not have done the same resulting in system hanging.

I confirmed this behavior today by deliberately missing the above step in compilation and it resulted in the same behavior as yesterday !

And that fork bomb explanation was nice in the link, thanks for that.

Quote:

Originally Posted by unSpawn

Right now the only thing one can say is that applying ulimits is always good. You did apply those, right?

What are ulimits ? I really don't know ! Perhaps I should RTFM.

druuna · 09-01-2010, 05:42 AM

Hi,

@anishakaul

Here you are: ulimit and sysctl - Linux Howto's

Hope this helps.

Aquarius_Girl · 09-01-2010, 05:48 AM

Quote:

Originally Posted by druuna

Here you are: ulimit and sysctl - Linux Howto's

Hope this helps.

You just pointed me to the link I was already reading, thanks anyway !

I tried this:

Code:

anisha@linux-uitj:~> ulimit -u 30
bash: fork: retry: Resource temporarily unavailable
bash: fork: retry: Resource temporarily unavailable
bash: fork: retry: Resource temporarily unavailable
bash: fork: retry: Resource temporarily unavailable
^C^C^C

^Cbash: fork: Resource temporarily unavailable

bash: fork: retry: Resource temporarily unavailable

bash: fork: retry: Resource temporarily unavailable
^Cbash: fork: retry: Resource temporarily unavailable

What does this mean ? and the Ctrl+c was not working !

The following also resulted in the same ! I changed 30 to 3 here !

Code:

anisha@linux-uitj:~> ulimit -u 3
bash: fork: retry: Resource temporarily unavailable
bash: fork: retry: Resource temporarily unavailable
bash: fork: retry: Resource temporarily unavailable
^C^Cbash: fork: retry: Resource temporarily unavailable

druuna · 09-01-2010, 06:12 AM

Hi,

This tool (ulimit) will not necessarily stop the initial process, it will limit the use of resource for a user or a specific terminal.

The message you get (also mentioned in the link I gave) tells you that a limit is reached. And even though that specific terminal might become unresponsive, other terminals and the machine itself are not compromised by, in this example, a bash fork bomb.

You could compare it with making a building fire proof: You can have a blazing fire in one room (which is destroyed, most of the time) but the adjacent rooms and building itself survive.

If I set max user processes to 30 and execute the fork bomb I see this:

Code:

$ ulimit -u 30
$ :(){ :|:& };:
bash: fork: Resource temporarily unavailable
bash: fork: Resource temporarily unavailable
[1] 17255
[1]+  Terminated              : | :

And, in this case, the process is gone as well. I do believe the termination depends a bit on which process creates the fork(s). I'm not sure, but I would understand if the following scenario where true: Process A forks a sub-process B. Process B starts forking multiple processes (loop, intentional or not). Ulimit kills Process B and its children when reaching the limit, but process A remains.

Hope this helps.