bash shell script read file line by line.

d.anitkumar · 10-03-2007, 05:15 AM

Hi This is my script like..
while read line
do
myval="DAKS"
done
echo $myval

i am not getting the value output outside the while loop
help me out if u havr some others

ghostdog74 · 10-03-2007, 06:34 AM

Quote:

Originally Posted by Darren[UoW]

I am using the following code to read line by line from a file, however my problem is that $value=0 at the end of the loop possibly because bash has created a subshell for the loop or something similar. How can I solve this.

value=0;

while read line
do
value=`expr $value + 1`;
echo $value;
done < "myfile"

echo $value;

Note: This example just counts the number of lines, I actually desire to do more complex processing than this though, so 'wc' is not an alternative, nor is perl im afraid.

Thanks Darren.

most of the problems i need to solve doesn't involve the use of loops at all. sometimes, using while loops sometime could be slower than just using tools like awk.
if you want to do more complex processing, why not try awk.

Code:

awk '{
     # some processing with file
     }
END { print "number of lines is " NR }' file

dpoper1 · 04-28-2008, 02:05 PM

hi,

I am trying to do a loop, the main puporse of the loop is that it will search in a multiple text files stored on tmp2 a pattern /remote-host so it would output the pattern into tmp3.

The code that I have is the following:

for i in $( tmp2 ) ; do
sed -n '/remote-host/p' tmp2 >> tmp3
done

and obviously is not working, I would appreciate any help.

regards,

mike

lbt · 08-10-2009, 06:39 AM

A simple solution to the subject line:

cat file |
(
while read line
do
echo process $line
done
)

ghostdog74 · 08-10-2009, 07:47 AM

you are digging up an old thread. check the date of post next time. also, no need to use cat. its useless

lbt · 08-10-2009, 08:14 AM

Quote:

Originally Posted by ghostdog74

you are digging up an old thread. check the date of post next time. also, no need to use cat. its useless

The only reason I posted is because such an old thread was the top search result on google and had no answers that a new user could, IMHO, understand.

Using cat rather than a redirect here (again IMO) makes the answer crystal clear and hopefully will educate rather than confuse

Code:

(
  while read line
  do
    echo process $line
  done
) < file

does work but it isn't clear as you start to read the code without realising that, potentially many lines later, a file will be directed into the sub shell.

If you are after an optimised solution feel free to use exec with a fd numeric redirect and read -u too :

Code:

exec 9<file
while read -u9 line
do
  echo process $line
done

Explanation left as an exercise for the reader...

thehog · 02-03-2010, 02:10 PM

Code:

#!/bin/bash

IFS=$'\12'

for i in $(cat somefile.txt); do 
  echo "line: $i";
done

kvmreddy · 05-15-2010, 02:18 AM

Check bellow link, there I posted 5 methods to parse a file line by line
Process a file line by line

catkin · 05-15-2010, 03:02 AM

Quote:

Originally Posted by kvmreddy

Check bellow link, there I posted 5 methods to parse a file line by line
Process a file line by line

The link is slightly mangled. Here's unmangled: Process a file line by line.

Nice blog

Regards request for correction ...

Regards "Bash can sometimes start a subshell in a PIPED "while-read" loop", what are the circumstances under which bash does not do so?

Regards

Code:

FILENAME=$1
...
done < $FILENAME

That will fail if $1 includes embedded IFS charcters; safer to use double quotes: done < "$FILENAME".

Regards "exec 3<&0 Now all of the keyboard and mouse input is going to our new file descriptor 3" it is only the keyboard input, not the mouse input.

Regards "while read LINE Using File Descriptor" an alternative, that does not require saving fd 0 and restoring it is to assign the file to, say, fd 3 and use read's -u option to read from fd3.

Regards typos, "The file descriptors for stdin,stdout, and stderr are 0,1, and 2, respectively" it is more correctly "The file descriptors for stdin, stdout, and stderr are 0, 1, and 2, respectively", that is with a space after the fist commas in the list.

The test timings are very useful showing how much faster awk is at processing a large file but it would be nice to see the other case -- the case when processing a single line. You would have to process the same line many times to get the timing and of course this would buffer both the input file and awk in RAM but it would still be interesting. I understand that the shell's fork-exec to run awk uses a lot of resource which will more than offset awk's much greater file IO and string handling efficiency.

kvmreddy · 05-17-2010, 07:19 AM

I have posted 5 different methods to process a file line by line in a shell script. check bellow link
Different ways to process a file line by line

tuxdev · 05-17-2010, 11:13 AM

Besides the quoting issue that catkin mentioned, you should also use read -r in all the examples. You almost never want to not use -r. Also, use "printf" instead of "echo -e". The exact behavior of "echo -e" can change a lot from system to system.

MTK358 · 05-17-2010, 12:41 PM

Why is everyone responding to this >6 year old thread?

Abid Malik · 10-16-2010, 09:46 AM

hello everyone!

i am passing an array in function. I want to read the first content again and again until it is empty so

function calltoarray
{
read $1[0]
while [ ! -n "$1[0]" ]
do
echo empty
read $1[0]
done
}
but the code is not working. please help me out.

sag47 · 10-20-2010, 06:04 AM

howto read file shell

Okay first off I realize this thread is old... very old. However the complexity that these guys are going about it is driving me nuts so I have to post this to enlighten them (and anyone who would be searching for this solution).

First off reading a file in shell is extremely simple.

lets make a simple file called mycat and let it's contents be...

Code:

#!/bin/bash
while read line;do
  echo $line
done

That will read from stdin so you can read a file by simply...

Code:

chmod 755 ./mycat
./mycat < somefiletoread

The other thing is this guy is basically going about it the hard way because this can easily be done with a one liner.

Quote:

Originally Posted by dpoper1

hi,

I am trying to do a loop, the main puporse of the loop is that it will search in a multiple text files stored on tmp2 a pattern /remote-host so it would output the pattern into tmp3.

The code that I have is the following:

for i in $( tmp2 ) ; do
sed -n '/remote-host/p' tmp2 >> tmp3
done

and obviously is not working, I would appreciate any help.

regards,

mike

And here it is...

Code:

find /path/to/tmp2 -type f -not -name 'tmp3' -print0 | xargs -0 grep -ih 'remote-host' > tmp3

Just thought I'd post in case someone else was dealing with a similar issue...

SAM

jthan24 · 03-14-2011, 02:20 PM

Hi, i test my script with a file and is excellent!!

for (( i = 0; i < `awk '{x++}END{ print x}' test.out`; i++))
do
echo `head -n $i test.out | tail -n 1` #return the line
done

where test.out is the file to scann!!