bash shell script read file line by line.

Ramurd · 11-06-2011, 02:04 AM

altered the script here like this:

Code:

#!/bin/bash

bylet()
{
	val=0
	iter=1000000
	for((i=0;i<iter;i++))
	do
		let val+=i	
	done

	echo ${val}
}

other()
{
	val=0
	iter=1000000
	for((i=0;i<iter;i++))
	do
		((val+=i))
	done
	echo ${val}
}
time bylet
time other

To reduce on the dereferences still got me the rather big difference of:
499999500000

real 0m10.552s
user 0m10.253s
sys 0m0.236s
499999500000

real 0m8.684s
user 0m8.413s
sys 0m0.234s

That's nearly 2 seconds; While the dereferencing saved 2 seconds; The difference got halved;
Changing the for loop to the old style reduced the time equally in both:

499999500000

real 0m7.705s
user 0m7.433s
sys 0m0.176s
499999500000

real 0m5.640s
user 0m5.542s
sys 0m0.089s

The difference stays 2 seconds in an iteration of 1 million simple calculations;
(( ))-style calculations are also built-in shell and in my belief generally a much stronger method.

Things get really weird when you add a little stuff:

Code:

#!/bin/bash

bylet()
{
	val=0
	iter=1000000
	for i in {0..999999}
	do
		let val+=i
		if [[ $val > 10000000 ]]
		then
			let val=5
		fi
	done

	echo ${val}
}

other()
{
	val=0
	iter=1000000
	for i in {0..999999}
	do
		((val+=i))
		if ((val > 10000000 )) 
		then 
			((val=5)) 
		fi
	done
	echo ${val}
}
time bylet
time other

MadeInGermany · 01-09-2012, 06:47 AM

Quote:

Originally Posted by Darren[UoW]

I am using the following code to read line by line from a file, however my problem is that $value=0 at the end of the loop possibly because bash has created a subshell for the loop or something similar. How can I solve this.

value=0;

while read line
do
value=`expr $value + 1`;
echo $value;
done < "myfile"

echo $value;

Note: This example just counts the number of lines, I actually desire to do more complex processing than this though, so 'wc' is not an alternative, nor is perl im afraid.

Thanks Darren.

Indeed a while loop is a block, that can be executed in a sub shell.
A sub shell is a fork of the main shell, so it inherits the variables, but there is no easy way to copy its variables back to the main shell.
It depends on the implementation:
Traditional Unix shells make it a sub shell when the block's input/output is redirected or piped.
Your work-around then is:

Code:

value=0
#save &0 in the unused &5
exec 5<&0
# read &0 from file
exec <"myfile"
while read line
do
value=...
...
done
#restore &0
exec <&5
echo $value

Posix shells, ksh, bash, zsh only make a sub shell when the block's input/output is piped (and they support the built-in $((numeric expression)).Darren's code works.

Ksh and zsh handle the last part of a pipe in the main shell, so even this one works:

Code:

value=0
cat "myfile" | while read line
do
value=...
...
done
echo $value

keirvt · 05-31-2012, 06:02 PM

#!/bin/ksh
# Using kshell so that the variable value is seen outside while loop
value=0;

while read line
do
value=`expr $value + 1`;
echo $value;
done < "myfile"

echo $value;

Ramurd · 05-31-2012, 11:13 PM

A few notes on that last post: I think the fipo was written by engraving letters in stone and somehow they've been able to digitize this.

Next thing: use code tags, like shown below.
Backticks are soooo 2011... use $() instead

That'd make your script look like this (changed a few more things - but essentially the same)

Code:

#!/bin/ksh
#Using Korn Shell to ease scoping of variables
value=0;

while read line
do
  ((value ++))
  printf "\r%05d" ${value}
done < "myfile"

printf "\nValue of \$value after the loop: %d\n" ${value}

Red_Lv · 12-12-2012, 12:11 AM

e...
are you sure that the script you posted show 0 at the end..
for i copy the code in my pc and it works as expected.

hfreyer · 07-04-2013, 04:08 AM

If you need to get whitespace at the beginning or end of the lines
you can locally modify IFS. Here is a small "wc" clone example:

Code:

cat baz
bar
  foo  
     foo bar

# Only leading blanks are used.

Code:

a=0; b=0; c=0; 
while IFS="" read -r line; do 
  a=$((a+1))            # line count
  set -- $(echo $line)
  b=$((b+${#@}))        # word count
  c=$((c+${#line}+1))   # char count
done < baz
echo "$a $b $c"
wc baz

output is:

Code:

3 4 23
 3  4 23 baz

konsolebox · 07-04-2013, 08:16 PM

Quote:

Originally Posted by Darren[UoW]

I am using the following code to read line by line from a file, however my problem is that $value=0 at the end of the loop possibly because bash has created a subshell for the loop or something similar. How can I solve this.
value=0;

while read line
do
value=`expr $value + 1`;
echo $value;
done < "myfile"

echo $value;

Quote:

I actually desire to do more complex processing than this though, so 'wc' is not an alternative, nor is perl im afraid.

And I expect that you don't need other scripting languages like Ruby as well? If so I think you just have to use Bash, as I see that you're probably using a different shell unless your file "myfile" is actually empty?. I've handled many shells already including the original bourne shell and I don't recall a shell that spawns a subprocess on a while loop unless that loop is reading a pipe but perhaps the very old ones does. Anyway if you have bash you could do it like this and it should be best enough:

Code:

#!/bin/bash

value=0

while read -r line; do  ## -r to prevent backslashes to escape characters.
    (( value += 1 ))
    echo "$value"  ## take note of the importance of placing string inside doublequotes especially when in a normal bash statement
done < "myfile"

echo "$value"

Norseman01 · 06-30-2015, 02:18 PM

[QUOTE='Darren[UoW];711668']I am using the following code to read line by line from a file, however my problem is that $value=0 at the end of the loop possibly because bash has created a subshell for the loop or something similar. How can I solve this.
.

value=0;

while read line
do
value=`expr $value + 1`;
echo $value;
done < "myfile"

echo $value;

-----------------------
try this:
wc -l file-in-question | cut --delimiter=" " -f1 -

It will return the first token (the line count) only.

To automate, one method is to put above wc command in a chmod 777 script named something like CountLns.scr
and change file-in-question to $1 and put CountLns.scr thefile in a loop that feeds it files.
examples:
CountLns.scr BirthdayPartyInvites.txt ==one shot
for f in *.txt; do CountLns.scr $f; done ==process current directory
for f in `find . -name "*.txt" -print`; CountLns.scr $f; done ==process current branch.
...You can sum the script's output, redirect it to another file, or whatever suits you.
.
Normally wc -l would return number of lines followed by file name.
Using CountLns.scr BirthdayPartyInvites.txt will return just 100 if that is the number of lines in that file.
. (removed greater than sign)
sh, wc and cut are stock aids in Linux.
don't forget to have first line of script start with #!/bin/bash
(or sh in place of bash if that is your flavor)

Norseman01

Norseman01 · 07-04-2015, 06:37 PM

Quote:

Originally Posted by Darren[UoW]

I am using the following code to read line by line from a file, however my problem is that $value=0 at the end of the loop possibly because bash has created a subshell for the loop or something similar. How can I solve this.
...(snip)

Note: This example just counts the number of lines, I actually desire to do more complex processing than this though, so 'wc' is not an alternative, nor is perl im afraid.

Thanks Darren.

================================
I useually use something like:

NLins=`wc $1 | cut - -d" " -f2`

--but you can put it in a wrapper to reduce typing.

#!/bin/bash
# Name: whatever you want to call it
# Purpose: Count Number of Lines in a (txt) file.
# by: Your_Name or Initials go here
# Date: should put at least month/year here
# Requireds: if any, they go here
# chmod 777 Whatever you named it
#
LineCount=`wc $1 | cut - -d" " -f2`
echo $LineCount
# end of file

but you can just type the "LineCount...." in wherever you please.
That includes from the command line.
(In which case substitute the actual filename,
complete with path if needed, in place of $1)

bash chmod, wc and cut are supplied verbs in Linux.
(see also man pages for bash, chmod, wc and cut.)

Norseman01

sag47 · 07-05-2015, 11:57 AM

Quote:

Originally Posted by Darren[UoW]

I am using the following code to read line by line from a file, however my problem is that $value=0 at the end of the loop possibly because bash has created a subshell for the loop or something similar. How can I solve this.

value=0;

while read line
do
value=`expr $value + 1`;
echo $value;
done < "myfile"

echo $value;

Note: This example just counts the number of lines, I actually desire to do more complex processing than this though, so 'wc' is not an alternative, nor is perl im afraid.

Thanks Darren.

How about asking your real question. Quit asking for people to solve steps to what you think is the correct solution. Tell us what your entire problem is and what you're trying to accomplish. If all you want to do is count lines in a file then use wc. If that's not what you're trying to do then stop beating around the bush and give us the real problem.

EDIT: this thread is 11 years old at this point. I don't know why people keep posting. It's confusing. Can a mod lock this thread?

jeremy · 07-09-2015, 12:46 PM

Quote:

Originally Posted by sag47

EDIT: this thread is 11 years old at this point. I don't know why people keep posting. It's confusing. Can a mod lock this thread?

The thread ranks highly as a search result, which is likely why this happened. If this continues to be an issue, I'll close the thread.

--jeremy

einer · 04-15-2016, 02:49 PM

#!bin/bash
##
#$1 is the file name to read
##
for i in `cat $1`; do
echo $i
done

Norseman01 · 04-17-2016, 06:07 PM

Quote:

Originally Posted by Darren[UoW]

I am using the following code to read line by line from a file,
...snip)
Thanks Darren.

=====================

The bottom line: end value is not right.

try this:

#!/bin/bash
#
lnno=0
#echo $lnno #use if line number increases AFTER processing.

while read aline
do
lnno=$(( $lnno + 1 )) # this line before or after processing
# processing # can make a difference. so make sure
# more processing # all commands are in sync just as the
# and more processing # same as with base 0 or base 1 in counting.
echo $lnno " " $aline # optional location just here as an example.
done <afile.txt
echo $lnno # should repeat last line number inside loop
# end of file
-----
Sample output of use with a text file. Binary files are best not printed direct.
Convert binaries to Hex or something before printing.

1
2
3 I was reading through the Complete CSS Guide and came to the Page Layout
4 Properties section. The comments on 'layer' are not definitive. It left
5 me confused. What was the author trying to say?
...
23 Computer Aided Drafting makes extensive use of LAYERS. Foundation,
24 floor, walls, wiring, piping, etc go on their respective LAYERS and for
25 good reasons.
26
27 A single element on a 2D plane is an element, not a layer. ...
... etc etc

now one can at least use the line numbers for conversation about a specific line in a text file.

Norseman01