[SOLVED] Init tries to respawn a running process

dafydd2277 · 08-20-2010, 01:12 PM

Good Morning,

This is on RHEL 5.4.

First clue: Three lines in /var/log/messages, Every 5 Minutes:

Code:

[Daemon name redacted]: Daemon started
last message repeated 9 times
init: Id "mp" respawning too fast: disabled for 5 minutes

Second clue: Run multiple ps commands. The daemon is running.

Code:

# ps -ef | grep [Redacted] | grep -v grep
root    21519     1  0 Jun08 ?  00:02:48 [Redacted]

Third clue: Re run the ps commands before and after one of these message triplets in /var/log/messages. The PID of the daemon does not change! Therefore, the process is not, in fact, being stopped and restarted.

Fourth clue: The relevant line in /etc/inittab:

Code:

mp:12345:respawn:/usr/sbin/[SubDir]/[Daemon]

Fifth clue: The execution line in the inittab entry is identical to the path and daemon name in the ps output's CMD field.

Problem: Why is init trying to respawn a running process? how do I tell it to stop making the inappropriate attempt, without removing the ability to restart the process when it ~does~ die?!

Thanks!
dafydd

PS: Occasionally, this daemon does die, which means we have to wait up to five minutes for it to respawn. In testing, the respawn is successful after the five minute wait, although I still get the "last message repeated 9 times" output with no decrease in the number of repeats. I have reports that the respawn fails, in that the daemon is never restarted. I've yet to recreate that problem.

eSelix · 08-20-2010, 01:42 PM

If pid hasn't changed then daemon wasn't restarted. You have time of working 00:02:48. Is this some standard daemon or yours? Maybe it fork or run another process and then quit, leaving child process (real daemon) working.

dafydd2277 · 08-20-2010, 03:16 PM

Hi, eSelix,

Turns out I got that same reply from one of the developers about half an hour before I saw this.

They've got a work around. I think I'll keep my "Workaround? Not a fix?" to myself.

dafydd

dafydd2277 · 08-10-2011, 10:41 AM

I'm going back over my old questions, and realized I never posted a "What did it turn out to be?" reply.

What it turned out to be is that the process being spawned by init launches just long enough to fork a copy of itself and then die.

So, init doesn't see the PID of the process it launched, and tries to launch another. The new one starts just long enough to realize the self-named child is still running, and then kills itself. So, init tries again. Lather, rinse, repeat. So, every five minutes, init goes through this process about 10 times in a second, and throttles itself for another five minutes.

The workaround from a year ago was to keep the parent process running but empty. I believe subsequent versions since then have done away with the fork.

Cheers!
dafydd