bash: feeding input to a script from a text file

faizlo · 11-17-2010, 10:02 AM

Hi all,

I have a script which checks on my jobs that run on some cluster.
The script, "script.sh", takes as an input the job-id for the job to be checked. Sometimes I have 100s of jobs and I want to check them all (for successful completion.) I could put these job-ids into a text file, "job-id.txt", each id in its own line.
For each job-id, the script would ask me few questions (with a yes or no answers) to see if I want to do some other checks for each job-id.

I want to know how may I direct my job-ids from this text file into the script one job-id at a time.

Thanks in advance,

~faizlo

GrapefruiTgirl · 11-17-2010, 10:10 AM

You could do something like this, using xargs:

Code:

xargs -a job-id.txt -I@ ./script.sh @

This provides similar functionality to doing a while/done loop in the shell, and feeding it the job-id.txt file with a re-direct:

Code:

while read line; do
 # "$line" is the job ID
 # ask questions here
done < job-id.txt

which is also (nearly) the same as doing it this way:

Code:

cat job-id.txt | while read line; do
 # "$line" is the job ID
 # ask questions here
done

druuna · 11-17-2010, 10:11 AM

Hi,

Something like this should work:

Code:

#!/bin/bash

for JOBID in `cat job-id.txt`
do

  --- your code goes here, $JOBID holds 1 entry ---

done

Hope this helps.

faizlo · 11-17-2010, 11:02 AM

@GrapefruiTGirl:
xargs yielded echoing the help message for xargs claiming that -a is not a valid option. I checked the man pages and -a is a valid option to feed from a file (instead of std input).
Adding the while loop in the script is a very good idea. I added my script here as I thought this may help because I did not know how/where to add the loop (sorry)

@druuna
I used your script as follows:

Code:

#!/bin/bash

for JOBID in `cat job-id.txt`
do

  ./script.sh $JOBID

done

And it worked fine. Thank you both so much for the quick help and wonderful ideas.

Well, your help honed my appetite to ask for more!

Since this script (script.sh) asks for job ids as an input, one at a time, I have to answer few questions with y or n.
How should I edit the script such that if any of the output messages contain: NO FILES TO RECOVER, it would discontinue checking and fall back to the next job-id?

here is my script:

Code:

check()
{
    echo "Do you want to proceed?"
    read answer
    if [ "${answer}" == "y" ] ; then
	echo "... proceeding ..."
    else
	echo "Aborting."
	exit 1
    fi
}


project=$1

echo "Make a recovery project: ${project}"
echo "First, look at the project summary."
check

echo "project summary --project=${project}"
project summary --project=${project}

echo "Now get the recovery dims."
check

echo "recovery project --printQuery --project=${project}"
dims=`recovery project --printQuery --project=${project}`

echo "$dims"
echo "Translate these dims."
check

echo "get --dims=\"${dims}\""
get --dims="${dims}"

echo "Create the recovery definition."
check

echo "definition --group=dzero --defName=recovery_project_${project} --dims=\"${dims}\""
definition --group=user --defName=recovery_project_${project} --dims="${dims}"

echo "Now check the definition."
#check

echo "get --summaryOnly --dims=\"__set__ recovery_project_${project}\""
get --summaryOnly --dims="__set__ recovery_project_${project}"

echo "DONE!"
echo "Recovery def: recovery_project_${project}"

thanks again

~faizlo

GrapefruiTgirl · 11-17-2010, 11:10 AM

Interesting.. I wonder why xargs -a did not work for you.. If you're interested, you could try the --long version of -a, which is --arg-file= instead of -a. So it would look like:

Code:

xargs --arg-file=job-id.txt -I@ ./script.sh @

Anyhow, even if this mysteriously does not work, I believe you're well along the way to a good solution. Let us know if any further troubles crop up, or you need more help.

faizlo · 11-17-2010, 11:27 AM

Yes, I still need some help.
Your loop idea is nice but I am not sure I know how should I add the loop into the script?

~faizlo

I tried xargs on both tcsh (which is my main shell, but not my shell of chice

) and on a bash terminal, both gave same complains about the -a option!
Also, the --arg-file= option did not work too!!! It threw the same help message.

GrapefruiTgirl · 11-17-2010, 11:46 AM

Code:

check()
{
    echo "Do you want to proceed?"
    read answer
    if [ "${answer}" == "y" ] ; then
	echo "... proceeding ..."
    else
	echo "Aborting."
	exit 1
    fi
}

while read project; do
#project=$1

echo "Make a recovery project: ${project}"
echo "First, look at the project summary."
check

echo "project summary --project=${project}"
project summary --project=${project}

echo "Now get the recovery dims."
check

echo "recovery project --printQuery --project=${project}"
dims=`recovery project --printQuery --project=${project}`

echo "$dims"
echo "Translate these dims."
check

echo "get --dims=\"${dims}\""
get --dims="${dims}"

echo "Create the recovery definition."
check

echo "definition --group=dzero --defName=recovery_project_${project} --dims=\"${dims}\""
definition --group=user --defName=recovery_project_${project} --dims="${dims}"

echo "Now check the definition."
#check

echo "get --summaryOnly --dims=\"__set__ recovery_project_${project}\""
get --summaryOnly --dims="__set__ recovery_project_${project}"

echo "DONE!"
echo "Recovery def: recovery_project_${project}"

done < job-id.txt

I made a couple small changes in your script above, and put them as bold text. This is probably not exactly how I would do this if I were writing this script for myself, but it demonstrates one way of implementing a loop that reads the items from the input file, and executes a bunch of code (everything between the while read and the done < blah blah.. parts) for each item.

So now, every time you run the above script, it will process the job-id.txt file, and loop through all entries inside it, asking you what to do. If you abort, then the entire process is aborted, so you might want to further adjust the code if you have the need to be able to skip a job, but not abort the whole process.

druuna · 11-17-2010, 11:46 AM

Hi,

Would this help?

Code:

#!/bin/bash
# -------------------------------------------------------------------------- #
# Variables
# ------------------------------------------------------------------ #

[[ "$1" == "" ]] && echo "syntax script.sh jobidfile" ; exit 1

project="$1"

# -------------------------------------------------------------------------- #
# Functions
# ------------------------------------------------------------------ #
check()
{
    echo "Do you want to proceed?"
    read answer
    if [ "${answer}" == "y" ] ; then
     echo "... proceeding ..."
    else
     echo "Aborting."
     exit 1
    fi
}


# -------------------------------------------------------------------------- #
# Main
# ------------------------------------------------------------------ #

for JOBID in `cat ${project}`
do

  echo "Make a recovery project: ${project}${JOBID}"
  echo "First, look at the project summary."
  check

  echo "DONE!"
  echo "Recovery def: recovery_project_${project}${JOBID}"

done

exit 0

# -------------------------------------------------------------------------- #
# End

I just placed the first and last commands you posted in post #4 inside the loop. So the above is now your script.sh

Hope this helps.

faizlo · 11-17-2010, 12:09 PM

@druuna
Shouldn't it be like this:

Code:

#!/bin/bash
# -------------------------------------------------------------------------- #
# Variables
# ------------------------------------------------------------------ #

[[ "$1" == "" ]] && echo "syntax script.sh jobidfile" ; exit 1

project="$1"

# -------------------------------------------------------------------------- #
# Functions
# ------------------------------------------------------------------ #
check()
{
    echo "Do you want to proceed?"
    read answer
    if [ "${answer}" == "y" ] ; then
     echo "... proceeding ..."
    else
     echo "Aborting."
     exit 1
    fi
}


# -------------------------------------------------------------------------- #
# Main
# ------------------------------------------------------------------ #

for JOBID in `cat ${project}`
do

  echo "Make a recovery project: $JOBID"
  echo "First, look at the project summary."
  check

  echo "DONE!"
  echo "Recovery def: recovery_project_$JOBID"

done

exit 0

# -------------------------------------------------------------------------- #
# End

druuna · 11-17-2010, 12:23 PM

Hi,

Yes, of course...... Stupid of me! Sorry.

tuxdev · 11-17-2010, 12:39 PM

Don't use the for file in `cat file` form, it's quite fragile and depends too much on what IFS happens to be at the time. The while-read form is *much* more robust. (also, $() is way better than backticks)

faizlo · 11-17-2010, 01:18 PM

Also, the first exit statement
exit 1 (at checking the input file statement) caused the whole script to exit with an exit status 1
I made a small adjustment:

Code:

#!/bin/bash

# -------------------------------------------------------------------------- #
# Variables
# -------------------------------------------------------------------------- #

if [ $# -lt 1 ]; then 
    echo "syntax make_recovery_project.sh job-id.txt";
    exit 1
fi

project="$1"


# -------------------------------------------------------------------------- #
# Functions
# -------------------------------------------------------------------------- #
check()
{
    echo "Do you want to proceed?"
    read answer
    if [ "${answer}" == "y" ] ; then
     echo "... proceeding ..."
    else
     echo "Aborting."
     exit 1
    fi
}


# -------------------------------------------------------------------------- #
# Main
# -------------------------------------------------------------------------- #

for JOBID in `cat ${project}`
do

echo "Make a recovery def for project:" $JOBID
echo "First, look at the project summary."
check

echo "get project summary --project="$JOBID
nofiles=`get project summary --project=$JOBID | grep -c \"NO FILES TO RECOVER\"`
if [ "$nofiles" != 0 ]; then
    echo "Nothing to do for this job, go to the next job ID ..."
    continue
fi

echo "Now get the recovery dims."
check

echo "generate strict recovery project --printQuery --project="$JOBID
dims=`generate strict recovery project --printQuery --project=$JOBID`

echo "$dims"
echo "Translate these dims."
check

echo "translate --dims=\"${dims}\""
translate --dims="${dims}"

echo "Create the recovery definition."
check

echo "create definition --group=dzero --defName=recovery_project_$JOBID --dims=\"${dims}\""
create definition --group=dzero --defName=recovery_project_$JOBID --dims="${dims}"

echo "Now check the definition."
#check

echo "translate --summaryOnly --dims=\"__set__ recovery_project_$JOBID\""
translate --summaryOnly --dims="__set__ recovery_project_$JOBID"

echo "DONE!"
echo "Recovery def: recovery_project_$"JOBID

done

exit 0

# -------------------------------------------------------------------------- #
# End

I will try it now!

~faizlo

faizlo · 11-17-2010, 01:21 PM

@tuxdev

Do you mean, instead of

Code:

for JOBID in `cat ${project}`

I should have

Code:

for JOBID in $(cat ${project})

?

grail · 11-17-2010, 08:07 PM

No he means instead of the for loop construct which is subject to word splitting based on IFS that you need to change it to the while loop construct
shown by GGirl, ie:

Code:

while read -r JOBID
do
    <your stuff here>
done< $project

I would also mention that your first test is not robust enough in that the user may also enter more than one parameter, which I believe is unacceptable. Plus
what if the file entered does not exist? Maybe we could using something like:

Code:

if (( $# != 1 )) && [[ ! -e "$1" ]]
then
    echo "syntax make_recovery_project.sh job-id.txt"
    exit 1
fi

faizlo · 11-18-2010, 01:17 PM

Hi again,

so I tried the loop part as suggested by GrapefruiTGirl and grail,
I could get this:

Code:

./recover_jobs.sh jobs-id.txt
Make a recovery def for project: 31980_20101114093600
First, look at the project summary.
Do you want to proceed?
Aborting.

which could read the first project id correctly but for some reason did not not ask me to answer y or anything else and just aborted!

Any hint?