LinuxQuestions.org
Page 4 of 4 « First 23 4
Show 50 post(s) from this thread on one page

LinuxQuestions.org (/questions/)
-   Programming (https://www.linuxquestions.org/questions/programming-9/)
-   -   bash shell script read file line by line. (https://www.linuxquestions.org/questions/programming-9/bash-shell-script-read-file-line-by-line-136784/)

Ramurd 11-06-2011 02:04 AM
altered the script here like this:
Code:
#!/bin/bash

bylet()
{
        val=0
        iter=1000000
        for((i=0;i<iter;i++))
        do
                let val+=i       
        done

        echo ${val}
}

other()
{
        val=0
        iter=1000000
        for((i=0;i<iter;i++))
        do
                ((val+=i))
        done
        echo ${val}
}
time bylet
time other
To reduce on the dereferences still got me the rather big difference of:
499999500000

real 0m10.552s
user 0m10.253s
sys 0m0.236s
499999500000

real 0m8.684s
user 0m8.413s
sys 0m0.234s

That's nearly 2 seconds; While the dereferencing saved 2 seconds; The difference got halved;
Changing the for loop to the old style reduced the time equally in both:

499999500000

real 0m7.705s
user 0m7.433s
sys 0m0.176s
499999500000

real 0m5.640s
user 0m5.542s
sys 0m0.089s

The difference stays 2 seconds in an iteration of 1 million simple calculations;
(( ))-style calculations are also built-in shell and in my belief generally a much stronger method.

Things get really weird when you add a little stuff:
Code:
#!/bin/bash

bylet()
{
        val=0
        iter=1000000
        for i in {0..999999}
        do
                let val+=i
                if [[ $val > 10000000 ]]
                then
                        let val=5
                fi
        done

        echo ${val}
}

other()
{
        val=0
        iter=1000000
        for i in {0..999999}
        do
                ((val+=i))
                if ((val > 10000000 ))
                then
                        ((val=5))
                fi
        done
        echo ${val}
}
time bylet
time other

MadeInGermany 01-09-2012 06:47 AM
Quote:
Originally Posted by Darren[UoW] (Post 711668)
I am using the following code to read line by line from a file, however my problem is that $value=0 at the end of the loop possibly because bash has created a subshell for the loop or something similar. How can I solve this.


value=0;

while read line
do
value=`expr $value + 1`;
echo $value;
done < "myfile"

echo $value;


Note: This example just counts the number of lines, I actually desire to do more complex processing than this though, so 'wc' is not an alternative, nor is perl im afraid.


Thanks Darren.
Indeed a while loop is a block, that can be executed in a sub shell.
A sub shell is a fork of the main shell, so it inherits the variables, but there is no easy way to copy its variables back to the main shell.
It depends on the implementation:
Traditional Unix shells make it a sub shell when the block's input/output is redirected or piped.
Your work-around then is:

Code:
value=0
#save &0 in the unused &5
exec 5<&0
# read &0 from file
exec <"myfile"
while read line
do
value=...
...
done
#restore &0
exec <&5
echo $value
Posix shells, ksh, bash, zsh only make a sub shell when the block's input/output is piped (and they support the built-in $((numeric expression)).Darren's code works.

Ksh and zsh handle the last part of a pipe in the main shell, so even this one works:

Code:
value=0
cat "myfile" | while read line
do
value=...
...
done
echo $value

keirvt 05-31-2012 06:02 PM
Problem easy solved by using kshesll
 
#!/bin/ksh
# Using kshell so that the variable value is seen outside while loop
value=0;

while read line
do
value=`expr $value + 1`;
echo $value;
done < "myfile"

echo $value;

Ramurd 05-31-2012 11:13 PM
A few notes on that last post: I think the fipo was written by engraving letters in stone and somehow they've been able to digitize this.

Next thing: use code tags, like shown below.
Backticks are soooo 2011... use $() instead

That'd make your script look like this (changed a few more things - but essentially the same)
Code:
#!/bin/ksh
#Using Korn Shell to ease scoping of variables
value=0;

while read line
do
  ((value ++))
  printf "\r%05d" ${value}
done < "myfile"

printf "\nValue of \$value after the loop: %d\n" ${value}

Red_Lv 12-12-2012 12:11 AM
the script works well in my pc
 
e...
are you sure that the script you posted show 0 at the end..
for i copy the code in my pc and it works as expected.

hfreyer 07-04-2013 04:08 AM
If you need to get whitespace at the beginning or end of the lines
you can locally modify IFS. Here is a small "wc" clone example:
Code:
cat baz
bar
  foo 
    foo bar
# Only leading blanks are used.
Code:
a=0; b=0; c=0;
while IFS="" read -r line; do
  a=$((a+1))            # line count
  set -- $(echo $line)
  b=$((b+${#@}))        # word count
  c=$((c+${#line}+1))  # char count
done < baz
echo "$a $b $c"
wc baz
output is:
Code:
3 4 23
 3  4 23 baz

konsolebox 07-04-2013 08:16 PM
Quote:
Originally Posted by Darren[UoW] (Post 711668)
I am using the following code to read line by line from a file, however my problem is that $value=0 at the end of the loop possibly because bash has created a subshell for the loop or something similar. How can I solve this.
value=0;

while read line
do
value=`expr $value + 1`;
echo $value;
done < "myfile"

echo $value;
Quote:
I actually desire to do more complex processing than this though, so 'wc' is not an alternative, nor is perl im afraid.
And I expect that you don't need other scripting languages like Ruby as well? If so I think you just have to use Bash, as I see that you're probably using a different shell unless your file "myfile" is actually empty?. I've handled many shells already including the original bourne shell and I don't recall a shell that spawns a subprocess on a while loop unless that loop is reading a pipe but perhaps the very old ones does. Anyway if you have bash you could do it like this and it should be best enough:
Code:
#!/bin/bash

value=0

while read -r line; do  ## -r to prevent backslashes to escape characters.
    (( value += 1 ))
    echo "$value"  ## take note of the importance of placing string inside doublequotes especially when in a normal bash statement
done < "myfile"

echo "$value"

Norseman01 06-30-2015 02:18 PM
line count for arbitrary file
 
[QUOTE='Darren[UoW];711668']I am using the following code to read line by line from a file, however my problem is that $value=0 at the end of the loop possibly because bash has created a subshell for the loop or something similar. How can I solve this.
.

value=0;

while read line
do
value=`expr $value + 1`;
echo $value;
done < "myfile"

echo $value;

-----------------------
try this:
wc -l file-in-question | cut --delimiter=" " -f1 -

It will return the first token (the line count) only.

To automate, one method is to put above wc command in a chmod 777 script named something like CountLns.scr
and change file-in-question to $1 and put CountLns.scr thefile in a loop that feeds it files.
examples:
CountLns.scr BirthdayPartyInvites.txt ==one shot
for f in *.txt; do CountLns.scr $f; done ==process current directory
for f in `find . -name "*.txt" -print`; CountLns.scr $f; done ==process current branch.
...You can sum the script's output, redirect it to another file, or whatever suits you.
.
Normally wc -l would return number of lines followed by file name.
Using CountLns.scr BirthdayPartyInvites.txt will return just 100 if that is the number of lines in that file.
. (removed greater than sign)
sh, wc and cut are stock aids in Linux.
don't forget to have first line of script start with #!/bin/bash
(or sh in place of bash if that is your flavor)

Norseman01

Norseman01 07-04-2015 06:37 PM
Quote:
Originally Posted by Darren[UoW] (Post 711668)
I am using the following code to read line by line from a file, however my problem is that $value=0 at the end of the loop possibly because bash has created a subshell for the loop or something similar. How can I solve this.
...(snip)

Note: This example just counts the number of lines, I actually desire to do more complex processing than this though, so 'wc' is not an alternative, nor is perl im afraid.


Thanks Darren.
================================
I useually use something like:

NLins=`wc $1 | cut - -d" " -f2`

--but you can put it in a wrapper to reduce typing.

#!/bin/bash
# Name: whatever you want to call it
# Purpose: Count Number of Lines in a (txt) file.
# by: Your_Name or Initials go here
# Date: should put at least month/year here
# Requireds: if any, they go here
# chmod 777 Whatever you named it
#
LineCount=`wc $1 | cut - -d" " -f2`
echo $LineCount
# end of file

but you can just type the "LineCount...." in wherever you please.
That includes from the command line.
(In which case substitute the actual filename,
complete with path if needed, in place of $1)

bash chmod, wc and cut are supplied verbs in Linux.
(see also man pages for bash, chmod, wc and cut.)

Norseman01

sag47 07-05-2015 11:57 AM
Quote:
Originally Posted by Darren[UoW] (Post 711668)
I am using the following code to read line by line from a file, however my problem is that $value=0 at the end of the loop possibly because bash has created a subshell for the loop or something similar. How can I solve this.


value=0;

while read line
do
value=`expr $value + 1`;
echo $value;
done < "myfile"

echo $value;


Note: This example just counts the number of lines, I actually desire to do more complex processing than this though, so 'wc' is not an alternative, nor is perl im afraid.


Thanks Darren.
How about asking your real question. Quit asking for people to solve steps to what you think is the correct solution. Tell us what your entire problem is and what you're trying to accomplish. If all you want to do is count lines in a file then use wc. If that's not what you're trying to do then stop beating around the bush and give us the real problem.

EDIT: this thread is 11 years old at this point. I don't know why people keep posting. It's confusing. Can a mod lock this thread?

jeremy 07-09-2015 12:46 PM
Quote:
Originally Posted by sag47 (Post 5387492)
EDIT: this thread is 11 years old at this point. I don't know why people keep posting. It's confusing. Can a mod lock this thread?
The thread ranks highly as a search result, which is likely why this happened. If this continues to be an issue, I'll close the thread.

--jeremy

einer 04-15-2016 02:49 PM
Bash, read a file line-by-line
 
#!bin/bash
##
#$1 is the file name to read
##
for i in `cat $1`; do
echo $i
done

Norseman01 04-17-2016 06:07 PM
Quote:
Originally Posted by Darren[UoW] (Post 711668)
I am using the following code to read line by line from a file,
...snip)
Thanks Darren.
=====================

The bottom line: end value is not right.


try this:

#!/bin/bash
#
lnno=0
#echo $lnno #use if line number increases AFTER processing.

while read aline
do
lnno=$(( $lnno + 1 )) # this line before or after processing
# processing # can make a difference. so make sure
# more processing # all commands are in sync just as the
# and more processing # same as with base 0 or base 1 in counting.
echo $lnno " " $aline # optional location just here as an example.
done <afile.txt
echo $lnno # should repeat last line number inside loop
# end of file
-----
Sample output of use with a text file. Binary files are best not printed direct.
Convert binaries to Hex or something before printing.

1
2
3 I was reading through the Complete CSS Guide and came to the Page Layout
4 Properties section. The comments on 'layer' are not definitive. It left
5 me confused. What was the author trying to say?
...
23 Computer Aided Drafting makes extensive use of LAYERS. Foundation,
24 floor, walls, wiring, piping, etc go on their respective LAYERS and for
25 good reasons.
26
27 A single element on a 2D plane is an element, not a layer. ...
... etc etc

now one can at least use the line numbers for conversation about a specific line in a text file.

Norseman01


All times are GMT -5. The time now is 10:41 PM.
Page 4 of 4 « First 23 4
Show 50 post(s) from this thread on one page