Is it possible to limit number of /sbin/sh instances???
I recently was forced to do a power button reboot on my server, basically, I couldnt ssh in with any username other than root and noticed that when I ran ps -A there were 100's of "sh" entries in there.
Also noticed my messages log shown multiple failed password attempts for root and the other usual suspects. so basically my mahicne had several hundred processes running and cpu usage was through the roof so I'm kind of thinking is there a way to limit the number of sh instances that can be opened and this might possibly prevent this going forward. Thanks as always... |
If you limit the number of sh processes you'd still have the same problem because at some point you wouldn't be able to login as your login opens a shell.
It sounds to me almost as if someone is existing the system improperly and leaving sh processes running. You can kill those with a "kill -1 <pid>" but I'd try to track down owners of the processes and find out how they're exiting the system. My guess is they're just closing windows or turning off workstations. |
I think you might be interested in the "ulimit" section of "man bash", maybe the -u option.
Quote:
|
Trouble killing processes
A few times now, I've had to reboot my server as a result of 100's of /bin/sh processes spawning, not entirely sure the cause of this yet but regardless, i'm having problems killing these processes.
using kill -9 PID or killall sh doesnt seem to remove any of them. Now I'm assuming I cant kill the init process, so without actually rebooting the machine, are there any other options open to me, until of course I find out why so many are spawning. Cant change to runlevel 1 or run the init 6, I'm forced to do a power off/on to get the server back to normal. Thanks in advance... |
you need to know at least how it was started? (What did this spawning started?) probably you can find the root process by parent pid or by name.
|
When I run pstree it seems to come from init.
Can't check anything else for now as server is down and I'm many miles away :( |
Also check "tail -n 40 /var/log/messages" and "dmesg | tail -n 40" for clues.
-- TTK |
@OP: I've merged your "Is it possible to limit number of /sbin/sh instances???" thread with this recent one as it is the same topic. Also note that you never responded to replies in that thread. If you did you might have solved or mitigated the problem over a month ago. Next to that it shouldn't just be one way traffic and any usable replies should warrant a response from you.
|
Sorry, I couldnt find that post actually, sorry and thanks for the heads up.
Update: I had a cron script that run every 5 minutes, basically I'm now thinking this may have been the cause of the problem and if so, just wont bother using it. I was using it to log ADSL connection drops. With regards to checking the logs, all logs after my logrotate were empty, shouldnt normally be the case I know. Boring bit, I didnt think as was posted above that limiting the number of sh instances running is actually what I was looking for which might have contributed to me not replying to the original post. My cry for help now is that when this happened again, I was unable to kill the offending processes and thats why I posted again. Sorry for any hassle and thanks for all interest and posts. |
A 'ps' I managed while the problem was there...
Code:
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND |
Quote:
|
Quote:
|
Ah - you are correct. I hadn't looked that deep. Well done to him then :-)
|
Quote:
Quote:
Do you run a standard logrotate configuration? Are all logs empty including all rotated ones? Does your syslog, cron or any other daemon log show any anomalies around the time of the log rotation? Are there any login (attempts) during or prior to this? Quote:
A few remarks if I may in random order: Quote:
- you seem to be logging in as root user. That is not a security best practice regardless of any seemingly mitigating arguments. Do use an unprivileged user account with pubkey auth to log in with. Quote:
Quote:
Quote:
|
Quote:
Quote:
From what I remember, only the secure/messages/maillog entries were empty, rotated ones were populated as expected. Additionally, I've killed webmin for the time being. I could only login as root, all other users simply hung after password entry from console. As always all comments are appreciated and I'm looking into the other points mentioned and when/if this happens again, I'll try and better gather the information to answer the questions I've been asked but as yet not been able to answer. O and this is 9.1 |
All times are GMT -5. The time now is 08:18 PM. |