Shell Script to Kill and Restart a process based on CPU load

kalisto · 12-15-2011, 09:12 PM

I have been having this problem with a mud i run, anytime it crashes the mud doesn't restart and sometimes it will hang and not allow anyone on, i was wondering if someone could help me tweak my startup script to make it so it kills the process when the cpu load gets around 40%~ and restarts it after killing it

here is what my startup script looks like right now

Code:

#! /bin/csh -f

# Set the port number.
set port = 8100
if ( "$1" != "" ) set port="$1"

# Change to area directory.
cd ~/dbzl/area

# Set limits.
limit coredumpsize unlimited
limit stacksize unlimited
if ( -e shutdown.txt ) rm -f shutdown.txt

while ( 1 )
    # If you want to have logs in a different directory,
    #   change the 'set logfile' line to reflect the directory name.
    set index = 1000
    while ( 1 )
        if ( $index > 1025 ) then
           rm -f ../log/*.log
           set index = 1000
        endif
        set logfile = ../log/$index.log
        if ( ! -e $logfile ) break
        @ index++
    end

    # Record starting time
    date > $logfile
    date > ../area/boot.txt

    # Run DBZL.
    # Check if already running
    set matches = `netstat -an | grep ":$port " | grep -c LISTEN`
    if ( $matches >= 1 ) then
        # Already running
        echo Port $port is already in use.
        exit 0
    endif
    ~/dbzl/src/dbzl $1 >&! $logfile

    # Restart, giving old connections a chance to die.
    if ( -e shutdown.txt ) then
        rm -f shutdown.txt
        exit 0
    endif
    sleep 5
end

Thanks in advance for anyones help

anomie · 12-16-2011, 03:00 PM

What OS / version?

Just some brainstorming for now:

You could make the process less nice(1).
You could enforce some limits using pam_limits(8).

I do not like the idea of a kludge that is constantly checking to see if it needs to restart a service. That's a sign that something is very wrong (and should be addressed in a more straightforward way, IMO).

kalisto · 12-18-2011, 08:09 PM

Its ubuntu 11.11 and the script wouldn't really bother me, im in the process of using a better codebase to update my work, so hopefully the problem will go away, but in the meantime i want a working script to keep it checking, its quite a big project to undertake and im already running into some walls there, but i will get them worked out eventually.

anomie · 12-20-2011, 06:33 PM

Try not to take the the wrong way, but - for starters - please script in Bourne shell, and not csh. You'll find more folks have experience with that (for good reason), and will be more able to help.

That said, there may be some difficulties in designing your script. The pseudocode looks something like this:

At regular intervals, check foo process's CPU usage.
If CPU usage < 40%, finish quietly.
If CPU usage >= 40%, kill foo process, and start foo service again.

Is that correct? If so, should foo be killed every time it's observed to be at 40% or above? (For instance, what if there is a very brief spike in CPU usage by foo just at the moment you're polling?)

Reuti · 12-21-2011, 05:59 AM

I’m wondering, what is meant by 40% in detail. A single process, even with nice 19 can get almost 100% of the CPU time. When there are enough other processes, it might never reach 40% at all, despite the fact that you would like to kill it.