bash child process

BinWondrin · 06-19-2008, 03:13 PM

Hi folks,
I regularly run scripts that access up to 100 servers to modify, query, install or whatever.
The script steps through a file containing the names of all the servers and then does its thing on one after another (using ssh sudo ...).
It happens sometimes that a server does not respond. In that case the script usually hangs.
I would like to run each server access as a kind of child process so that the script doesn't die just because one server out of a bunch did not respond.
Maybe I should spawn child processes, keep track of the PID and kill them if they don't finish within a certain amount of time.

Here is part of my script template:

Code:

until [ $done ]
do
read <&3 servername
        if [ $? != 0 ]; then
                done=1
                continue
        fi
        RESULT1=`ssh $servername "sudo command goes here " 2> /dev/null`
        RESULT2=`ssh $servername "sudo command goes here " 2> /dev/null`
        echo $RESULT1 $RESULT2 $servername
done

Any ideas, sample code etc. would be welcome.

Just BinWondrin

colucix · 06-19-2008, 03:18 PM

I usually ping the remote server before attempting any operation:

Code:

if ping -c1 -W1 $servername > /dev/null
then
    echo remote server is up and running
else
    echo remote server is down
fi

just to illustrate.

ntubski · 06-19-2008, 04:08 PM

Quote:

Maybe I should spawn child processes, keep track of the PID and kill them if they don't finish within a certain amount of time.

That sounds like a good idea, as a bonus if none of servers are down, you would get a x100 speed up, on the other hand, I think you would have to collect the output into a file, or do the processing in the child process. The ping option is certainly easier as long the server doesn't fail after the ping but before the ssh command finishes.

Also I think that loop could be rewritten as

Code:

while read servername ; do
        RESULT1=`ssh $servername "sudo command goes here " 2> /dev/null`
        RESULT2=`ssh $servername "sudo command goes here " 2> /dev/null`
        echo $RESULT1 $RESULT2 $servername
done <&3

BinWondrin · 06-19-2008, 04:45 PM

Thanks for the replies.
Unfortunately I cannot ping the servers because most of them are firewalled and will only accept traffic on port 22.

BinWondrin · 06-19-2008, 05:01 PM

I tried that shorter cleaner loop but it only reads the 1st servername, executes the commands and exits.
Here is part of my echotime script. I wrote this initially to check if all servers are reachable before I run any other scripts. Kind of like the "ping" idea mentioned above.

I modified the loop as you suggested but it reads only the first servername before it exits. What am I missing?

Code:

# read til the end of the file
while read servername ; do
    printf '%15s' "$servername"
    /usr/bin/time -o temp.$$ -f %e ssh $servername exit 2> /dev/null
    RESULT1=`awk ' { print $1 } ' temp.$$`
    COMPARE=`awk ' { print $1 * 1000 } ' temp.$$`
    let "DASHES"="($COMPARE / 25) - 2"
    printf '%10s' "$RESULT1   |"
    cnt=0
    while [ $cnt -lt $DASHES ]; do
        printf "-"
        let cnt=$cnt+1
    done
    printf '>\n'
done <&3

unSpawn · 06-19-2008, 05:07 PM

Quote:

Originally Posted by BinWondrin

Maybe I should spawn child processes, keep track of the PID and kill them if they don't finish within a certain amount of time.

You could let the child kill itself:

Code:

child() { function killSelf() { sleep ${sleep:=10s}; kill $$; }; killSelf & disown %1; $someCommand; }

Quote:

Originally Posted by BinWondrin

Unfortunately I cannot ping the servers because most of them are firewalled and will only accept traffic on port 22.

Try netcat:

Code:

nc -w 2 -z remoteHost 22 >/dev/null && doSomething || dont

BinWondrin · 06-19-2008, 05:25 PM

Hey UnSpawn,

I am really impressed by the "nc" option. This is a function that I was marginally aware of but it did not really occur to me to use it. It just wasn't part of my day-2-day toolkit.
I love it! I checked out the man page for "nc" and I can already see all kinds of uses for it.
So, thank you for pointing that one out to me. It will certainly be used.

I have to wrap my mind around the child() function. I will try it out and if I can make it work it will be way quicker than what I am doing now.

osor · 06-19-2008, 05:26 PM

Quote:

Originally Posted by BinWondrin

Thanks for the replies.
Unfortunately I cannot ping the servers because most of them are firewalled and will only accept traffic on port 22.

You have a few options:

You can “ping” on port 22 using netcat or similar.
You might change the ConnectTimeout option if you are using OpenSSH.
You can use the following construct:
Code:
```
RESULT1=`ssh $servername "sudo command goes here " 2> /dev/null & sleep 10 && kill %1`
```
The reason this works is that you already start a subshell with “``”.

Edit: Wow, I need to refresh more often. 3 posts before I hit submit!

BinWondrin · 06-19-2008, 05:34 PM

All of them GREAT ANSWERS.

I am really impressed with this forum!

It is a very active forum and the quality of the replies is very high. I am very glad that I joined, folks.

I hope I can pay back somehow.

So thanks again for all of your answers.

Got to go and improve a bunch of scripts!

BinWondrin ( and now I know!)

colucix · 06-19-2008, 06:25 PM

Quote:

Originally Posted by BinWondrin

I modified the loop as you suggested but it reads only the first servername before it exits. What am I missing?

For further reference, this is a known issue with ssh. By default it tries to read from the standard input and sends a SIGTTIN signal (stop tty input) when exits. If the ssh command is inside a while loop taking input from stdin, the SIGTTIN stops the input and prevents further processing. To avoid this behaviour use the -n option:

Code:

ssh -n user@host

which explicitly redirects stdin from /dev/null, leaving the standard input of the loop untouched.