LinuxQuestions.org
Support LQ: Use code LQ3 and save $3 on Domain Registration
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices

Reply
 
Search this Thread
Old 02-22-2011, 03:40 PM   #1
Neruocomp
Member
 
Registered: Oct 2004
Distribution: Slackware, CentOS
Posts: 135

Rep: Reputation: 15
Parse text for host list


To run a parallel chemistry program, I need to build the host list. The cluster already has SGE(grid engine) installed and it generates a host list file with the following contents:

compute-1-1.local 2 all.q@compute-1-1.local UNDEFINED
compute-2-1.local 2 all.q@compute-1-1.local UNDEFINED

The important bits are hostname(ex compute-1-1) and number of cpus to use(ex 2). And for this program, it wants them in this form, a shell variable: HOSTLIST=hostname:cpus=X hostname:cpus=X ....

I've tried this script, but it doesn't work
Code:
#!/bin/bash
spacer=":cpus="

let count=0

cat hostfile | while read line; do
  HOSTS[$count]=$line
  ((count++))
done

NNODES=${#HOSTS[@]}

for ((i=0;i<$NNODES;i++)); do
  host=`echo ${HOSTS[$i]}|cut -f1 -d" "`
  cpus=`echo ${HOSTS[$i]}|cut -f2 -d" "`
  HOSTS[$i]="$host$spacer$cpus"
done

for j in ${HOSTS[@]}  
do
  HOSTLIST=$HOSTLIST+$HOSTS[$j]
done
Any ideas? Thanks
 
Old 02-22-2011, 04:06 PM   #2
Tinkster
Moderator
 
Registered: Apr 2002
Location: in a fallen world
Distribution: slackware by choice, others too :} ... android.
Posts: 22,988
Blog Entries: 11

Rep: Reputation: 880Reputation: 880Reputation: 880Reputation: 880Reputation: 880Reputation: 880Reputation: 880
Not sure I got it right, but:
Code:
export HOSTLIST=$( awk '{printf "%s:cpus=%s ",$1, $2}' hostlist )
echo $HOSTLIST 
compute-1-1.local:cpus=2 compute-2-1.local:cpus=2


Cheers,
Tink
 
Old 02-22-2011, 04:45 PM   #3
Neruocomp
Member
 
Registered: Oct 2004
Distribution: Slackware, CentOS
Posts: 135

Original Poster
Rep: Reputation: 15
Hey that worked like a charm. I guess I gotta just sit down and read up on awk >_< BTW can no one tell me why my script doesn't work?
 
Old 02-22-2011, 05:45 PM   #4
Tinkster
Moderator
 
Registered: Apr 2002
Location: in a fallen world
Distribution: slackware by choice, others too :} ... android.
Posts: 22,988
Blog Entries: 11

Rep: Reputation: 880Reputation: 880Reputation: 880Reputation: 880Reputation: 880Reputation: 880Reputation: 880
Quote:
Originally Posted by Neruocomp View Post
Hey that worked like a charm. I guess I gotta just sit down and read up on awk >_< BTW can no one tell me why my script doesn't work?


It *does* work. Your only problem is that outside your script
HOSTLIST is unknown.



Cheers,
Tink
 
Old 02-23-2011, 11:43 AM   #5
Neruocomp
Member
 
Registered: Oct 2004
Distribution: Slackware, CentOS
Posts: 135

Original Poster
Rep: Reputation: 15
No, this would have ended up part of a larger script where HOSTLIST would have been referenced. The thing was while testing this snip of code, I would put in echo statements to watch the variables and NNODES would come up as zero and I don't know why.
 
Old 02-23-2011, 01:39 PM   #6
Tinkster
Moderator
 
Registered: Apr 2002
Location: in a fallen world
Distribution: slackware by choice, others too :} ... android.
Posts: 22,988
Blog Entries: 11

Rep: Reputation: 880Reputation: 880Reputation: 880Reputation: 880Reputation: 880Reputation: 880Reputation: 880
Quote:
Originally Posted by Neruocomp View Post
No, this would have ended up part of a larger script where HOSTLIST would have been referenced. The thing was while testing this snip of code, I would put in echo statements to watch the variables and NNODES would come up as zero and I don't know why.
Hmmm ... it worked here, w/ echoes strewn in.

Maybe your file had DOS line endings or something, as opposed to what you
copy & pasted here?
Code:
~/tmp$ time ./host.sh 

compute-1-1.local:cpus=2 compute-2-1.local:cpus=2

real    0m0.013s
user    0m0.006s
sys     0m0.004s
Code:
~/tmp$ time export HOSTLIST=$( awk '{printf "%s:cpus=%s ",$1, $2}' hostlist )

real    0m0.005s
user    0m0.002s
sys     0m0.002s
~/tmp$ echo $HOSTLIST 
compute-1-1.local:cpus=2 compute-2-1.local:cpus=2

Cheers,
Tink
 
  


Reply

Tags
awk, shell


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
'E:Malformed line 54 in source list /etc/apt/sources.list (dist parse), E:The list of vsssuccess@gmail.com Linux - Desktop 1 11-17-2010 08:17 AM
how to find a least busy host given a hardcoded host list? justnil Linux - Networking 1 10-12-2010 10:53 AM
parse text between html wakatana Programming 4 10-27-2009 08:12 AM
How to parse text file to a set text column width and output to new text file? jsstevenson Programming 12 04-23-2008 02:36 PM
host: parse of /etc/resolv.conf failed D.A. Linux - Software 4 08-09-2007 12:30 PM


All times are GMT -5. The time now is 07:49 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration