ssh across a bunch of nodes

podollb · 04-29-2004, 01:34 PM

Hi,
I have a network of computers I am using as a cluster (worker nodes). I need to start a process on each one and make it long lived. I know of the "nohup" prefix, but if I have 100 nodes and I want to ssh to all the nodes in an automated way, how can I do that... So basically run a script like: startWorkers.sh or something that will logon to each one and start a worker process and then logout.

Matt Collier · 04-29-2004, 01:39 PM

#!/bin/sh
hosts="host1 host2 host3 ..."
for i in $hosts; do
ssh login@$i startWorkers.sh
done

Blinker_Fluid · 04-29-2004, 02:11 PM

You need PDSH.
http://www.llnl.gov/linux/pdsh/pdsh.html

It allows you to run commands across all the nodes in your cluster or just specified nodes. I believe it also comes with pdcp that allows you to copy files around your cluster.

podollb · 04-29-2004, 02:19 PM

That sounds like it doesn't work to well under ssh (especially if passwords are needed), and if I was using rsh I wouldn't use that, a script to start jobs across rsh is trivial.

Matt Collier · 04-29-2004, 03:09 PM

passwordless ssh is easy enough to configure if you trust the machines involved

Blinker_Fluid · 04-29-2004, 03:10 PM

I sure hope you aren't putting in a password to ssh to 100 nodes in a cluster.

You do have ssh set up on your cluster so you don't have to put a password right?

The advantage of pdsh is you are doing all the nodes at once. The script earlier in this thread is going to do things in a serial fashion. I believe you can compile it to run via ssh but just like rsh you have to set up the nodes so you don't have to put in a password.

podollb · 04-29-2004, 03:26 PM

I guess the clusters I have used always had rsh so it was simple to use, but my current problem is that the cluster (is really a network of computers) and it doesn't have ssh. So your above options sound good, but I don't know too much about how to use passwordless ssh, besides ssh-agent and using an ssh key (is that what you are referring to?)

Matt Collier · 04-29-2004, 03:40 PM

that is what i'm referring to, yes, though if you already know how to do it, i'm afraid i don't understand your 2nd comment about passwords not making things work.

besides, if ssh isn't an option, then it a moot point really.

pdsh looks far better anyway, since it'll manage the connections for you, though if it were me, i'd still rather use pdsh over ssh, particularly if you need to send the same passwords out over and over on an untrusted network

podollb · 04-29-2004, 03:47 PM

Yeah I do know about passwordless I guess and pshd might be an option...
But I do konw that pshd is only used with passwordless (at the prompt anyway) ssh.

For my problem it doesn't matter if things are started sequentially or not since I am basically starting worker nodes that are listening for work...

This may be a silly question but with the sequential script above how do I get around the script waiting at the first host (until that process terminates)? I tried putting a & at the end of my command but to no avail...

Matt Collier · 04-29-2004, 03:54 PM

'daemonise' the script, fork the process and make the parent exit, then set the group of the child to 0, and make sure you have some kind of exit handler for the children to clean up after themselves

#!/usr/bin/perl

my $childpid = fork;
exit if $childpid;
setpgrp(0,$$);

podollb · 04-29-2004, 07:24 PM

Could you give me a simple example? I am not clear where I would start my program with your perl script...

podollb · 05-11-2004, 03:35 PM

I got this to work for my needs:

#!/bin/csh

foreach host (`cat /etc/bhosts`)
echo "***** ${host} *****"
rsh ${host} $* &
end