Looping in Many Files with Shell Programming

ali2011 · 07-26-2012, 01:21 PM

I have a file_one that contains 200 IPs, on which I execute the following command to be broken down to 2 IPs per column:

Code:

awk '{a[NR%2+1]=a[NR%2+1]" "$0}  END {for (i in a){print a[i]}}' file_one > file_two

As a result, each column of file_two contains two IPs, and I save each column using awk to 100 files of names ip_list# (i.e., the # sign goes from 1 to 100). After that I run the following script on these 100 files so each file will have its own terminal.

Code:

while read machine; do
echo "  ";
echo "# Outer loop: $machine"; 
echo "=============================";
ssh -n -i /home/.ssh/key user_one@$machine 'myCommand';
done < ip_list#

Is there another way to perform this task in more convenient manner so the above script will loop through all the 100 files simultaneously from a single terminal (i.e., one call to the script)?

divyashree · 07-27-2012, 06:31 AM

If you want to perform task simultaneously from a single terminal with ssh, then use pssh(parallel ssh)/cssh(clustered ssh).

e.g.

Quote:

pssh -h <host-file> date

where the host-file contains the list of the hosts/ips.

Reuti · 07-27-2012, 07:09 AM

What’s the problem with running your script will the original file? Do you want to have 2 machines handled in each terminal window instead in one loop across all 200 machines?

ali2011 · 07-27-2012, 11:52 AM

Hi divyashree,

Your suggestion seems to work, but for each node I have to provide an RSA-key, username in order to access. I tried the command below, but it gives the following error:

Code:

./parallel-ssh -h q5 -A -l user_name myCommand

Code:

Traceback (most recent call last):
File "./parallel-ssh", line 115, in <module>
hosts = psshutil.read_hosts(opts.host_files, default_user=opts.user)
File "/usr/lib/pymodules/python2.6/psshlib/psshutil.py", line 18, in read_hosts
f = open(pathname)
IOError: [Errno 2] No such file or directory: 'q5

I tried many formats I found online, but no success. Do you know what is the correct format for q5 should be? And please check if the command is not correct.

ali2011 · 07-27-2012, 12:00 PM

Hi Reuti,

The script above has no error; it just a matter of simplicity. I need to ssh an X-number of machines every time, say 100 out of 200 simultaneously from one terminal. Clearly, I ask 100 out of 200 nodes to measure certain metric at once using single terminal, and then move to the next set of nodes remaining. I'm looking for some thing, on which I can have control on the number of nodes to be ssh-ed simultaneously.

ali2011 · 07-27-2012, 01:30 PM

Now I'm having the first pssh command working correctly, but the second when I try to pass my shell script, it times out although the script is working fine!

The first:

Code:

pssh -h q1 -l -P user_name "cd ping/final; /sbin/ifconfig;"

The second:

Code:

pssh -h q1 -l user_name "cd ping/final; chmod +x shell.sh;./shell.sh"