SH: Passing a variable out of a "while ; do ; done" loop

frenchn00b · 07-03-2011, 03:56 AM

Hello SH fan Coders,

"While ; do ; done" is very convenient for SH coding. However sometimes you may be annoyed by your computed variable within the "while do done" type loop.
What to do how to pass it out of the loop to the outside of the bash code? A solution is to write it into the /tmp or on the disk... and to call it back after.

- not elegant... really not...

Anyone would know a trick another alternative that would look nicer?

Thanks in advance ...

Code:

			# Count file total size
			TOTAL_SIZE=0
			LISTOFFILES=`cat "$HOME/.fvwmoscfg/fvwmburnerlist.lst"`
			echo "$LISTOFFILES"  | while read i ; do 
				SIZE=`du -bs $i | cut -f 1`
				TOTAL_SIZE=`expr $SIZE + $TOTAL_SIZE`
				echo "$TOTAL_SIZE" > "$HOME/.fvwmoscfg/fvwmburnerlisttotalsize.lst"
			done
			TOTAL_SIZE=`cat $HOME/.fvwmoscfg/fvwmburnerlisttotalsize.lst`
			echo "The total size of all files and folders is : $TOTAL_SIZE"

grail · 07-03-2011, 05:56 AM

Your issue is down to the fact that you feed your loop through a subshell (ie you pipe the data in). Try feeding the values / files directly into the loop with
redirection:

Code:

while read i
do
    <your stuff here>
done<"$HOME/.fvwmoscfg/fvwmburnerlist.lst"

You should now find all variables from within the loop are available outside

frenchn00b · 07-23-2011, 03:30 PM

Thank you it works !!

Try this script and see if it gives you any clues:

Code:

echo "---" >tmp
while read line; do
ps aux | grep "$0" | grep -v grep
done <tmp
rm -f tmp

MTK358 · 07-23-2011, 04:00 PM

Quote:

Originally Posted by frenchn00b

Try this script and see if it gives you any clues:

Code:

echo "---" >tmp
while read line; do
ps aux | grep "$0" | grep -v grep
done <tmp
rm -f tmp

???

grail · 07-23-2011, 04:16 PM

Quote:

Try this script and see if it gives you any clues

I am with MTK ... why would it give me any clues ... you asked the question? Also, the script you have presented makes little sense.

Nominal Animal · 07-23-2011, 04:56 PM

I am assuming your SH in the thread title means you are using a POSIX shell, and don't want to use e.g. Bash features.

The solution is obvious when the subshell does not need to output anything to standard output - for example,

Code:

#!/bin/sh

TOTAL=$(
	SIZE=0
	for FILE in "$@" ; do
		SIZE="$(du -bs "$FILE" | cut -f 1) + $SIZE"
	done
	expr $SIZE
)
echo "Total size: $TOTAL bytes."

I assume frenchn00b and everybody else is quite aware of that.

When the subshell needs to write to standard output, use a sub-subshell, and redirect its standard output to an extra descriptor. Use yet another descriptor to output to the original standard output.

In other words, swap standard output with an extra descriptor for a sub-subshell, then swap them back in the subshell. That way the sub-subshell can write to standard output and standard error normally, but if it wants to write to the original output, it has to use the extra descriptor.

Because we cannot swap descriptors in POSIX shells, we need two temporary extra descriptors. The easiest is to open the two extra descriptors in the outer shell, but you can of course reuse them any number of times. (Alternatively, you can use another subshell.)

Code:

#!/bin/sh

# Use 9 as the result descriptor,
# and 8 as the temporary output descriptor.
exec 8>&1 9>&1

TOTAL=$( (
		SIZE="0"
		for FILE in "$@" ; do
			TEMP=$(du -bs "$FILE" | cut -f 1)
			echo "$FILE: $TEMP bytes"
			SIZE="$TEMP + $SIZE"
		done
		expr $SIZE >&9

	) 9>&1 >&8 )

# The descriptors 8 and 9 can now be reused.

echo "Total: $TOTAL bytes"

You can of course use a pipe, not just a simple subshell:

Code:

#!/bin/sh

# Use 9 as the result descriptor,
# and 8 as the temporary output descriptor.
exec 8>&1 9>&1

TOTAL=$(
	for FILE in "$@" ; do
		echo "$FILE"
	done | (
		SIZE="0"
		while read FILE ; do
			TEMP=$(du -bs "$FILE" | cut -f 1)
			echo "$FILE: $TEMP bytes"
			SIZE="$TEMP + $SIZE"
		done
		expr $SIZE >&9
	) 9>&1 >&8 )

# The descriptors 8 and 9 can now be reused.

echo "Total: $TOTAL bytes"

When you have more than one stream of data, you will have to use files.

There is a trick to temporary files too. In fact, using this trick, I prefer temporary files over the pipe swap above.

Instead of using individual temporary files (named in separate variables), create a safe temporary directory, and tell the POSIX shell to remove it automatically when the script exits. That way, you never need to worry about cleaning it up, and you can use any number of temporary files -- and use descriptive names for them, too.

Code:

#!/bin/sh

# Create a safe autodeleted temporary directory.
WORK=$(mktemp -d) || exit $?
trap "rm -rf '$WORK'" EXIT

# Your script can now use any file under "$WORK",
# for example "$WORK/file-list" or "$WORK/status"
# as a temporary file, and never worry about
# removing any of them. The shell will remove
# it automatically when this script exits.

Please note the quoting used for the trap. Because the outermost quotes are double quotes, $WORK will be evaluated immediately, and not when the trap triggers. This is important! This way, even if you unset or change $WORK you will not affect the trap at all, it will still remove the original directory. I find this extremely easy to use.

I hope some of you find this useful.

MTK358 · 07-23-2011, 05:38 PM

Quote:

Originally Posted by Nominal Animal

Code:

#!/bin/sh

...

$( ... )

...

I thought that $(...) is bash-specific syntax.

tuxdev · 07-23-2011, 10:00 PM

Quote:

I thought that $(...) is bash-specific syntax.

It's actually POSIX, though some /bin/sh are actually Bourne and don't support it (damn you, Solaris).

Nominal Animal · 07-24-2011, 03:54 PM

Yup, it is in POSIX. If you want the temporary directory to work with practically all shells, just replace $(...) with `...` -- I think even Bourne shells have trap ... EXIT support. At least the SunOS 5.10 one does.

Note that the descriptor swap trick often gets complicated with backticks, since backticks cannot be nested. (POSIX does allow it if you escape the inner pair(s), but the ancient /bin/sh do not support that.) Complicated is not impossible, though. You simply need to use a function to replace the nested backtick expression. This is the most complicated example above, but rewritten to work with both POSIX shells, and ancient /bin/sh's:

Code:

#!/bin/sh

setSIZES () {
    SIZES=`LANG=C LC_ALL=C ; ls -ldG "$@" | awk '{ s+=$4 } END { print s }'`
    export SIZES
}

# Use 9 as the result descriptor,
# and 8 as the temporary output descriptor.
exec 8>&1 9>&1

TOTAL=`
	for FILE in "$@" ; do
		echo "$FILE"
	done | (
		SIZE="0"
		while read FILE ; do
			setSIZES "$FILE"
			echo "$FILE: $SIZES bytes"
			SIZE="$SIZES + $SIZE"
		done
		expr $SIZE >&9
	) 9>&1 >&8
       `

# The descriptors 8 and 9 can now be reused.

echo "Total: $TOTAL bytes"

Basically, setSIZES runs ls in a subshell, then uses awk to sum the file sizes. On SunOS 5.10 at least, du does not support the -b (size in bytes) option. The sum is saved in SIZES in the caller's environment. (It will not propagate to parent shells, so the value is only available in the immediate caller's environment. If you were to use the function inside backticks, the value is also available within the backticked expression. Like I said: complicated.)

Finally, the above scripts were not by any means the optimal way to count file sizes at all. They just illustrate the swap-output-descriptor communications method. For counting file sizes in bytes portably, I'd use something like

Code:

#!/bin/sh

if [ ":$*" = ":" ] || [ ":$*" = ":-h" ] || [ ":$*" = ":--help" ]; then
    exec >&2
    echo ""
    echo "Usage: $0 [ -h | --help ]"
    echo "       $0 files .."
    echo ""
    exit 0
fi

LANG=C
LC_ALL=C
export LANG LC_ALL

ls -ldGF "$@" | awk '
  ($0 !~ /\/$/ && NF > 7) {
    size=$4
    name=$0
    sub(/^[\t ]*[^\t ]+[\t ]+[^\t ]+[\t ]+[^\t ]+[\t ]+[^\t ]+[\t ]+[^\t ]+[\t ]+[^\t ]+[\t ]+[^\t ]+[\t ]/, "", name)
    sub(/[*\/>|@=]$/, "", name)
    total+=size
    printf("%12.0f\t%s\n", size, name)
  }
  END {
    printf("%12.0f\n", total)
  }'

Since ls -ldGF is fairly portable, producing the same output in most OSes when using the C locale, the above should be fairly reliable. It will b0rk if the user name contains a space, or if the file name contains a newline, though. It does work for both SunOS 5.10 /bin/sh (a Bourne shell) and dash-0.5.5.1 (a POSIX shell). The awk scriptlet uses the sub() function to remove the fields prior to the file name, because otherwise any whitespaces in the file name will be lost. (This should works with gawk, nawk, mawk, and even xpg4 awk.)
This is to say that if something feels overly complicated, there usually is some other tool you can use instead. Here, I replaced du with ls for compatibility reasons, and used awk for both the processing and output. It is much faster than any shell loop, too.

frenchn00b · 11-08-2011, 11:57 AM

Quote:

Originally Posted by grail

Your issue is down to the fact that you feed your loop through a subshell (ie you pipe the data in). Try feeding the values / files directly into the loop with
redirection:

Code:

while read i
do
    <your stuff here>
done<"$HOME/.fvwmoscfg/fvwmburnerlist.lst"

You should now find all variables from within the loop are available outside

thank you it worked great.

I did before VARIABLE=` cat "$HOME/.fvwmoscfg/fvwmburnerlist.lst"`

would you know how to implement variable into the command above you gave me, so that we do not need to write on hdisk?

grail · 11-09-2011, 01:58 AM

Quote:

so that we do not need to write on hdisk?

Please explain what you mean by the above statement?

David the H. · 11-09-2011, 02:09 PM

In a posix shell you can use a heredoc for input, expanding the variable(s) inside it.

Code:

#!/bin/sh

text='foo
bar
baz
bum'

while read line; do

     echo "$line"

done <<LIMITSTRING
$text
LIMITSTRING

You can use command substitutions too, of course, such as `cat *.txt`.

Reuti · 11-09-2011, 03:23 PM

Quote:

Originally Posted by David the H.

Code:

#!/bin/sh

text='foo
bar
baz
bum'

while read line; do

     echo "$line"

done <<LIMITSTRING
$text
LIMITSTRING

Variables can also be used directly by a triple <:

Code:

#!/bin/sh

text='foo
bar
baz
bum'

while read line; do

     echo "$line"

done <<<"$text"

David the H. · 11-10-2011, 12:48 AM

Quote:

Originally Posted by Reuti

Variables can also be used directly by a triple <:

True, except that this discussion is specifically about posix-compliant scripts. The here string is a bash extension, whereas here documents are supported everywhere.

http://mywiki.wooledge.org/HereDocument

See here for a good description of the subshell problem and its workarounds:

http://mywiki.wooledge.org/BashFAQ/024