variables inside a shell script

dsids · 01-02-2011, 06:51 AM

Hi,

Please corrent me if I am wrong.

In a shell script, the scope of a variable defined inside a loop (say 'for loop') is limited to the loop, that is, the value ceases to exist. If that is true, is it possible for me to pass the value of a variable outside of the loop when the loop exits.

For eg

for loop{
some iteration happening here....
...val=10...
}

echo $val

$val should print 10

Thanks

catkin · 01-02-2011, 06:59 AM

You are wrong

Code:

#!/bin/bash

for ((i=0; i<5; i++))
do
    [[ $i -eq 3 ]] && var=foo
done

echo $var

The output is foo.

dsids · 01-02-2011, 08:20 AM

Quote:

Originally Posted by catkin

You are wrong

Code:

#!/bin/bash

for ((i=0; i<5; i++))
do
    [[ $i -eq 3 ]] && var=foo
done

echo $var

The output is foo.

Thanks..but my script isn't printing the value:

it goes like this:

while loop
{
statement
if loop
{
statement
if loop
{
statement
else
statement
for loop
{
iteration
}
val1
val2
fi # first if ends here
fi #second if ends here

done

echo $val2 #is not printing here

catkin · 01-02-2011, 09:08 AM

What sort of shell script is that? Is that the exact code or pseudo-code?

Nominal Animal · 01-02-2011, 12:11 PM

There are a number of shells available for linux, with quite different syntaxes.

Catkin is quite right: in Bash (and in all other shells I know of) the variables defined in a loop are available after the loop.

I suspect you might have a totally different problem: Variables defined in a sub-shell are not available outside it. Example:

Code:

#!/bin/bash
value=1
( value=2
echo "subshell: value=$value"
)
echo "value=$value"

This outputs

Code:

subshell: value=2
value=1

Note that in addition to "()", the right side of a pipe "|" is also a subshell.

There are a number of ways to get around this problem. An easy and portable way is to use a temporary file; this works especially well if there are more than one item of data. The only tricky part is how to create the temporary files securely and portably. Here is an example:

Code:

#!/bin/bash

# Safely create a temporary directory for our temporary files.
TEMPS="/tmp/`hostname -f`.$$"
trap 'rm -rf "$TEMPS"' EXIT
if ! mkdir -m 0700 "$TEMPS" ; then
echo "$TEMPS: Cannot create temporary directory." >&2
exit 1
fi

# Use $TEMPS/value to transfer variable 'value'.
value=1
echo "value=$value"
( echo "2" > "$TEMPS/value" )
read value < "$TEMPS/value"
echo "After reading new value: value=$value"

# Another method: Source the temporary file.
# This is very versatile, but dangerous; be careful!
echo "some=$some, other=$other"
true > "$TEMPS/definitions"
( echo "some=5" >> "$TEMPS/definitions"
echo "other='Blah'" >> "$TEMPS/definitions"
)
. "$TEMPS/definitions"
echo "After sourcing the file, some=$some, other=$other"

I recommend using a temporary directory, because directory creation is an atomic operation (even over NFS); there is no race conditions to worry about, and with -m 0700 only the user is allowed to access it. As fully-qualified domain name followed by the process ID is unique, it is very good choice for the directory name, even if /tmp were shared on multiple machines. Finally, the trap command makes sure the entire temporary directory and its contents are removed when the script exits. Note that "true > file" is an easy way to make sure "file" exists and is empty. This avoids errors if the subshell does not create or empty the file.

The above script was written for Bash, but it is adaptable to almost all other shells with a little work.

Another possibility is to pipe the data to the outer shell, using a file descriptor:

Code:

#!/bin/bash

value=$RANDOM
echo "Before subshell: value=$value"
value=`( echo In the subshell.
echo Outputting 2 to descriptor 3 ..
echo 2 >&3
echo Done.
) 3>&1 1>&2`
echo "After subshell: value=$value"

Here, the standard output of the subshell is redirected to standard output, then descriptor 3 to standard output; because standard output is assigned to variable value, value gets assigned whatever the subshell outputs to descriptor 3. The order of the redirections is important (and counter-intuitive!), see Bash Reference Manual on redirections for details. However, since not all shells provide redirection, this method is only suitable for some shells.

Hope this helps,

Nominal Animal

dsids · 01-03-2011, 01:30 AM

My exact problem is I want to find out the total size of bunch of similar files on a given date...I am using bash on a Solaris

cat /apps/cdf/$dir/emmgsmdir.txt| while read line
do

ls -l /apps/cdf/log/$line.20101228.LOG > /dev/null 2>&1 #checks to see whether the process is running on that day, if it is then it created a log file as above

if [ $? -eq 0 ]
then

proc=`grep -i cdfProc /apps/cdf/log/$line.20101228.LOG| tail -1|awk '{print $NF}'` # greps directory where I will find the files required to calculate the size

if [ "$proc" == "" ]
then

echo "Proc directory not found"

else

procdir=`dirname $proc`
cd $procdir

echo
echo "I am in `pwd`"

k="0"
sumtotal="0"
for j in `ls -ltr | grep "$dte" | awk '{print $5}'` #this loop is to find each file and cuts out the size
do
k=`expr $k + $j` # each file size is added
done

echo "Size in Bytes: $k"

echo "Size in MB: `expr $k / 1048576`"

sumtotal=`expr $sumtotal + $k`
fi
fi
# I want to define a variable which calculates the total size of similar files echoes it to the screen. Is there any other place I can define a variable

done

grail · 01-03-2011, 02:07 AM

Is there a reason you are writing this script and not just using find? Or is this practice?

Also, please place code in [code][/code] tags so it is at least slightly readable.

dsids · 01-03-2011, 02:30 AM

Quote:

Originally Posted by grail

Is there a reason you are writing this script and not just using find? Or is this practice?

Also, please place code in [code][/code] tags so it is at least slightly readable.

Sorry about that, I did not know about the CODE tags.

Code:

cat /apps/cdf/$dir/emmgsmdir.txt| while read line
do


ls -l /apps/cdf/log/$line.20101228.LOG > /dev/null 2>&1 #checks to see whether the process is running on that day, if it is then it created a log file as above


if [ $? -eq 0 ]
then

proc=`grep -i cdfProc /apps/cdf/log/$line.20101228.LOG| tail -1|awk '{print $NF}'` # greps directory where I will find the files required to calculate the size

if [ "$proc" == "" ]
then

echo "Proc directory not found"

else

procdir=`dirname $proc`
cd $procdir

echo
echo "I am in `pwd`"

k="0"
sumtotal="0"
for j in `ls -ltr | grep "$dte" | awk '{print $5}'` #this loop is to find each file and cuts out the size
do
k=`expr $k + $j` # each file size is added
done

echo "Size in Bytes: $k"

echo "Size in MB: `expr $k / 1048576`"

sumtotal=`expr $sumtotal + $k`
fi
fi
# I want to define a variable which calculates the total size of similar files echoes it to the screen. Is there any other place I can define a variable

done

I went through the find man page on Solaris but I am unable to figure out how find can help me

I wrote a script as I figured I would need to grep specific files and add their sizes, which would require a loop.

Plus there a tens of directories, each with a different pattern, under which there are hundreds of different files, again some similar and some not.

The whole structure made me go for a script

Appreciate your help

grail · 01-03-2011, 04:11 AM

So I have had a look at the code and I have a number of issues

I will start from the top and work my way down:

1. You can simply read the file into the while loop using redirection

Code:

done</apps/cdf/$dir/emmgsmdir.txt

The advantage of this over cat is that if you make changes to the script inside the loop and need to use that information outside it will not be there due to pipe creating a subshell

2. This one goes to everywhere you have used ls ... don't ... see here for details.

3. Assuming the script is run on different days, hard coding the date will be an issue. Simply use date function:

Code:

LOG=/apps/cdf/log/$line.$(date '+%Y%M%d').LOG

4. The use of ls and then checking the return code is not required as bash provides a test for the existence of a file:

Code:

if [[ -f /apps/cdf/log/$line.20101228.LOG ]]

# or if you use LOG above
if [[ -f $LOG ]]

5. The following can be simplified to use only one external application:

Code:

proc=`grep -i cdfProc /apps/cdf/log/$line.20101228.LOG| tail -1|awk '{print $NF}'`

# becomes

proc=$(awk 'BEGIN{IGNORECASE=1}/cdfproc/{x=$NF}END{print x}' $LOG)

6. Testing a variable is of zero length has its own test:

Code:

if [[ -z $proc ]]

7. Creation of procdir not required as it is never used again and you can get the same result without external call but using parameter substitution (assuming your bash version supports other look at option 2)

Code:

# Option 1
cd ${proc%/*}

# Option 2
cd $(dirname $proc)

Now I would mention here that you also never test to see if this directory exists before continuing??

8. Why create number based variables as strings?? eg. k="0" ... why not just .. k=0

9. $dte does not exist anywhere else in the presented code so grep will fail

10. There is no test inside loop to differentiate between files and directories or other types??

11. Assuming grep does not fail, use the right tools for the job of arithmetic:

Code:

((k+=j))

This goes for the sumtotal later on as well

12. sumtotal will always be the same as k total as it is reset to zero (0) inside the loop

That is the end of the critic on the code you have so far. Please take positively that I am trying to help

As for find, I do see that the directories to be looked in will be those pointed to by $LOG (from my examples), but this is still doable.
Something like:

Code:

find ${proc%/*} -maxdepth 1 -type f -printf "%s\n" | awk '{sum+=$0}END{print sum}'

So you could assign this to k and away you go

dsids · 01-03-2011, 05:19 AM

Quote:

Originally Posted by grail

So I have had a look at the code and I have a number of issues

I will start from the top and work my way down:

1. You can simply read the file into the while loop using redirection

Code:

done</apps/cdf/$dir/emmgsmdir.txt

The advantage of this over cat is that if you make changes to the script inside the loop and need to use that information outside it will not be there due to pipe creating a subshell

2. This one goes to everywhere you have used ls ... don't ... see here for details.

3. Assuming the script is run on different days, hard coding the date will be an issue. Simply use date function:

Code:

LOG=/apps/cdf/log/$line.$(date '+%Y%M%d').LOG

4. The use of ls and then checking the return code is not required as bash provides a test for the existence of a file:

Code:

if [[ -f /apps/cdf/log/$line.20101228.LOG ]]

# or if you use LOG above
if [[ -f $LOG ]]

5. The following can be simplified to use only one external application:

Code:

proc=`grep -i cdfProc /apps/cdf/log/$line.20101228.LOG| tail -1|awk '{print $NF}'`

# becomes

proc=$(awk 'BEGIN{IGNORECASE=1}/cdfproc/{x=$NF}END{print x}' $LOG)

6. Testing a variable is of zero length has its own test:

Code:

if [[ -z $proc ]]

7. Creation of procdir not required as it is never used again and you can get the same result without external call but using parameter substitution (assuming your bash version supports other look at option 2)

Code:

# Option 1
cd ${proc%/*}

# Option 2
cd $(dirname $proc)

Now I would mention here that you also never test to see if this directory exists before continuing??

8. Why create number based variables as strings?? eg. k="0" ... why not just .. k=0

9. $dte does not exist anywhere else in the presented code so grep will fail

10. There is no test inside loop to differentiate between files and directories or other types??

11. Assuming grep does not fail, use the right tools for the job of arithmetic:

Code:

((k+=j))

This goes for the sumtotal later on as well

12. sumtotal will always be the same as k total as it is reset to zero (0) inside the loop

That is the end of the critic on the code you have so far. Please take positively that I am trying to help

As for find, I do see that the directories to be looked in will be those pointed to by $LOG (from my examples), but this is still doable.
Something like:

Code:

find ${proc%/*} -maxdepth 1 -type f -printf "%s\n" | awk '{sum+=$0}END{print sum}'

So you could assign this to k and away you go

Thanks a lot. You're a genius..I still haven't gone through the exact recommendations that you have given as I need to read it well and then I'm sure you're gonna be bugged by a hundred questions. Thanks a lot for your help and patience

I am going through it right now...

dsids · 01-03-2011, 08:15 AM

Quote:

Originally Posted by grail

So I have had a look at the code and I have a number of issues

I will start from the top and work my way down:

1. You can simply read the file into the while loop using redirection

Code:

done</apps/cdf/$dir/emmgsmdir.txt

Code:

echo "Please enter dir name"
read dir


while read line
do

if [[ -f /apps/cdf/log/$line.20101228.LOG ]]
then


proc=$(awk 'BEGIN{IGNORECASE=1}/cdfproc/{x=$NF}END{print x}' /apps/cdf/log/$line.20101228.LOG)

        if [[ -z $proc ]]
        then
                echo "Proc dir not found"
        else
                cd $proc


                sum=`ls -ltr | grep "Dec 28" | awk -F" " 'BEGIN { sum=0 } { sum+=$5 }END { print sum }'`

                total=`expr $sum / 1048576`

                echo "$total"

        fi
fi


done</apps/cdf/$dir/emmgsmdir.txt

I modified the script but it is echoing 'Proc dir not found' when actually there is a proc dir in certain log files

Can you help me by explaining what that awk statement does?

From what I think..it is ignoring the case and grepping cdfproc where x is the last field and at the end it is printing x

grail · 01-03-2011, 08:36 AM

Have another look at point 7 that I made and see that it is not the same as the cd command you have in your else.

Nominal Animal · 01-04-2011, 03:47 PM

Quote:

Originally Posted by dsids

I went through the find man page on Solaris but I am unable to figure out how find can help me

You need to create two files, $AFTERFILE just before the desired period (day), and $NOTAFTERDATE at the end of the desired period (day). It's easiest to compute the timestamps needed with GNU awk. Consider the following example script. It takes zero or more directories as a parameter, followed by the year, month, and day as numbers, as command-line parameters:

Code:

#!/bin/bash

# Directories to search are listed first; otherwise just the working directory.
WHERE=()
while [ $# -gt 3 ]; do
WHERE=("${WHERE[@]}" "$1")
shift 1
done
[ ${#WHERE[@]} -gt 0 ] || WHERE=(".")

# Read command line parameters as "YEAR MONTH DAY", numbers only!
YEAR=$[ $1 -0 ] ; [ $YEAR -lt 1970 ] && YEAR=$[2000 + $YEAR]
MONTH=$[ $2 -0 ] ; [ ${#MONTH} -eq 1 ] && MONTH="0$MONTH"
DAY=$[ $3 -0 ] ; [ ${#DAY} -eq 1 ] && DAY="0$DAY"

if ! AFTERDATE=`gawk -v d="$YEAR $MONTH $DAY 00 00 -1" 'BEGIN { t=mktime(d); if (t==-1) exit(1); print strftime("%Y%m%d%H%M.%S", t); }'` ; then
echo "Cannot understand the date, sorry." >&2
exit 1
fi
if ! NOTAFTERDATE=`gawk -v d="$YEAR $MONTH $[DAY+1] 00 00 -1" 'BEGIN { t=mktime(d); if (t==-1) exit(1); print strftime("%Y%m%d%H%M.%S", t); }'` ; then
echo "Cannot understand the date, sorry." >&2
exit 1
fi

# Create timestamp files for use with 'find -newer':
AFTERFILE="/tmp/after.`hostname -f`.$$"
NOTAFTERFILE="/tmp/notafter.`hostname -f`.$$"
# Remove the temporary files whenever this script exits.
trap 'rm -f "$AFTERFILE" "$NOTAFTERFILE"' 0
# Create the temporary files with the desired modification timestamps.
touch -m -t "$AFTERDATE" "$AFTERFILE"
touch -m -t "$NOTAFTERDATE" "$NOTAFTERFILE"

# This would find all files modified on the desired day:
# find "${WHERE[@]}" -type f -newer "$AFTERFILE" '!' -newer "$NOTAFTERFILE"

# This outputs the total size of files modified on the desired day:
find "${WHERE[@]}" -type f -newer "$AFTERFILE" '!' -newer "$NOTAFTERFILE" -exec stat --printf '%s\n' '{}' ';' 2>/dev/null | gawk '{s+=$1} END { printf("%.6f MiB\n", s/1048576.0) }'

Using GNU/Linux, which has a better 'find' and a 'date' which understands things like "2 days ago", the same script is just

Code:

#!/bin/bash

WHERE=()
while [ $# -gt 1 ] && [ -d "$1" ]; do
WHERE=("${WHERE[@]}" "$1")
shift 1
done
[ ${#WHERE[@]} -gt 0 ] || WHERE=(".")

if ! AFTERDATE=`date -ud "$*" +'%Y%m%d' 2>/dev/null` ; then
echo "Cannot understand date '$*', sorry." >&2
exit 1
fi
NOTAFTERDATE=$[`date -ud "$*" +'%s' 2>/dev/null` + 24*60*60 + 3]
if ! NOTAFTERDATE=`date -ud "@$NOTAFTERDATE" +'%Y%m%d' 2>/dev/null` ; then
echo "Cannot understand date '$*', sorry." >&2
exit 1
fi

find "${WHERE[@]}" -type f -newermt "$AFTERDATE" "!" -newermt "$NOTAFTERDATE" -printf '%s\n' | gawk '{s+=$1} END { printf("%.6f MiB\n", s/1048576.0) }'

(The +24*60*60+3 at calculating the $NOTAFTERDATE is to add "at least one day", even if UTC time was adjusted (maximum allowed is +2 seconds, I believe). +36*60*60 would work just as well.)

Both Solaris and GNU/Linux 'find' support limiting the search to specific depths (0 at $WHERE, 1 = one directory down from $WHERE, and so on) using -mindepth N and -maxdepth N, if you put them before the other parameters.

Hope this helps,

Nominal Animal