Parse text for host list

Neruocomp · 02-22-2011, 03:40 PM

To run a parallel chemistry program, I need to build the host list. The cluster already has SGE(grid engine) installed and it generates a host list file with the following contents:

compute-1-1.local 2 all.q@compute-1-1.local UNDEFINED
compute-2-1.local 2 all.q@compute-1-1.local UNDEFINED

The important bits are hostname(ex compute-1-1) and number of cpus to use(ex 2). And for this program, it wants them in this form, a shell variable: HOSTLIST=hostname:cpus=X hostname:cpus=X ....

I've tried this script, but it doesn't work

Code:

#!/bin/bash
spacer=":cpus="

let count=0

cat hostfile | while read line; do
  HOSTS[$count]=$line
  ((count++))
done

NNODES=${#HOSTS[@]}

for ((i=0;i<$NNODES;i++)); do
  host=`echo ${HOSTS[$i]}|cut -f1 -d" "`
  cpus=`echo ${HOSTS[$i]}|cut -f2 -d" "`
  HOSTS[$i]="$host$spacer$cpus"
done

for j in ${HOSTS[@]}  
do
  HOSTLIST=$HOSTLIST+$HOSTS[$j]
done

Any ideas? Thanks

Tinkster · 02-22-2011, 04:06 PM

Not sure I got it right, but:

Code:

export HOSTLIST=$( awk '{printf "%s:cpus=%s ",$1, $2}' hostlist )
echo $HOSTLIST 
compute-1-1.local:cpus=2 compute-2-1.local:cpus=2

Cheers,
Tink

Neruocomp · 02-22-2011, 04:45 PM

Hey that worked like a charm. I guess I gotta just sit down and read up on awk >_< BTW can no one tell me why my script doesn't work?

Tinkster · 02-22-2011, 05:45 PM

Quote:

Originally Posted by Neruocomp

Hey that worked like a charm. I guess I gotta just sit down and read up on awk >_< BTW can no one tell me why my script doesn't work?

It *does* work. Your only problem is that outside your script
HOSTLIST is unknown.

Cheers,
Tink

Neruocomp · 02-23-2011, 11:43 AM

No, this would have ended up part of a larger script where HOSTLIST would have been referenced. The thing was while testing this snip of code, I would put in echo statements to watch the variables and NNODES would come up as zero and I don't know why.

Tinkster · 02-23-2011, 01:39 PM

Quote:

Originally Posted by Neruocomp

No, this would have ended up part of a larger script where HOSTLIST would have been referenced. The thing was while testing this snip of code, I would put in echo statements to watch the variables and NNODES would come up as zero and I don't know why.

Hmmm ... it worked here, w/ echoes strewn in.

Maybe your file had DOS line endings or something, as opposed to what you
copy & pasted here?

Code:

~/tmp$ time ./host.sh 

compute-1-1.local:cpus=2 compute-2-1.local:cpus=2

real    0m0.013s
user    0m0.006s
sys     0m0.004s

Code:

~/tmp$ time export HOSTLIST=$( awk '{printf "%s:cpus=%s ",$1, $2}' hostlist )

real    0m0.005s
user    0m0.002s
sys     0m0.002s
~/tmp$ echo $HOSTLIST 
compute-1-1.local:cpus=2 compute-2-1.local:cpus=2

Cheers,
Tink