Visit Jeremy's Blog.
Go Back > Forums > Linux Forums > Linux - Newbie
User Name
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!


  Search this Thread
Old 01-24-2007, 08:43 AM   #1
Registered: Jun 2006
Location: Galway
Distribution: Suse 10.1
Posts: 114

Rep: Reputation: 15
Arrow how to loop over text file lines within bash script for loop?


I have a data file containing 4 rows of data say:

X1 Y1 Z1
X2 Y2 Z2
X3 Y3 Z3
X4 Y4 Z4

Associated with each individual row is a data file containing a subset of additional data for that row. These associated data files are single column files, but not all have the same number of columns. For example that data file associated with row 1 might be of form:


Is there a way that I can pass the data file above (R1a, etc) to a for loop within a bash script such that it loops over the lines of the data file and outputs the following:

X1 Y1 Z1 R1a
X1 Y1 Z1 R1b
X1 Y1 Z1 R1c
X1 Y1 Z1 R1d

- and so on for each record of the main data file.

I've seen examples with individual files of directories being passed into for loops, as in:

for i in *

But what is the equivalent code when you want to loop over text file lines?

Old 01-24-2007, 09:22 AM   #2
LQ Guru
Registered: Sep 2003
Location: Bologna
Distribution: CentOS 6.5 OpenSuSE 12.3
Posts: 10,509

Rep: Reputation: 1978Reputation: 1978Reputation: 1978Reputation: 1978Reputation: 1978Reputation: 1978Reputation: 1978Reputation: 1978Reputation: 1978Reputation: 1978Reputation: 1978
There are many possible solutions. Here's mine, using gawk inside a while loop:



while [ $i -le `wc -l filename.txt | gawk '{print $1}'` ] ; do

    line=`head -$i filename.txt | tail -1`
    gawk -v prefix="$line" '{print prefix,$0}' $i.txt >> output.txt

    i=`expr $i + 1`
where filename.txt is the datafile and I assume that each "single column" file is called 1.txt (associated with first row) 2.txt (associated with second row) and so on. If the real "single column" files have progressive numbers in their name, you can easily build the name inside the loop. Otherwise you can do a list of these files and parse this list with the head/tail construct above.

Hope this will help.
Old 04-30-2009, 07:28 PM   #3
LQ Newbie
Registered: Apr 2009
Posts: 1

Rep: Reputation: 0
Wink Much easier

Set the internal file separator:

IFS=$'\n'; for line in `ls`; do echo $line; done
(this just gets each line for you easier than head/tail. the rest is the same)

Last edited by scriptjunkie1; 04-30-2009 at 07:30 PM.
Old 04-30-2009, 07:50 PM   #4
Senior Member
Registered: Aug 2006
Posts: 2,697
Blog Entries: 5

Rep: Reputation: 244Reputation: 244Reputation: 244
this is an old thread.
first, no need to change IFS and using ls is "useless" .
for line in *
secondly, OP wants to loop over the lines in the files (based on his post's last sentence), not filenames.
lastly, the tool to use could be simple paste/join.
1 members found this post helpful.
Old 05-01-2009, 06:11 AM   #5
LQ Guru
Registered: Oct 2005
Location: $RANDOM
Distribution: slackware64
Posts: 12,928
Blog Entries: 2

Rep: Reputation: 1287Reputation: 1287Reputation: 1287Reputation: 1287Reputation: 1287Reputation: 1287Reputation: 1287Reputation: 1287Reputation: 1287
Why not just use paste ?

paste file1 file2 > file 3
1 members found this post helpful.
Old 05-01-2009, 08:27 AM   #6
Registered: Apr 2009
Location: ~
Distribution: RHEL, Fedora
Posts: 381

Rep: Reputation: 58
Try Something like this

exec 3>&0
exec 0<$filename
while read eachline
echo $eachline
exec 0>&3

Old 03-20-2011, 03:48 PM   #7
LQ Newbie
Registered: Feb 2011
Posts: 1

Rep: Reputation: 0
Use cat

Better late than never

I was looking for the same answer and saw this post.
Eventually, I came up with this:

for line in `cat fileName.txt`
Old 03-20-2011, 07:38 PM   #8
Registered: May 2008
Posts: 107

Rep: Reputation: 21
My favorite:

while read line; do
    echo "$line"
done < "$filename"
Old 03-20-2011, 07:38 PM   #9
Registered: May 2008
Posts: 107

Rep: Reputation: 21
My favorite:

while read line; do
    echo "$line"
done < "$filename"
1 members found this post helpful.
Old 04-05-2011, 09:18 AM   #10
Registered: Apr 2011
Location: Philadelphia
Posts: 158

Rep: Reputation: 3
Nice work
Old 07-28-2015, 03:49 PM   #11
Registered: Apr 2015
Location: Yorkshire, England
Distribution: Linux Mint
Posts: 53

Rep: Reputation: Disabled
Originally Posted by mrm5102 View Post
Nice work
Whos done this nice work the OP doesn't ask for many derivatives of read every line in a group of files first line of all files, second line of all files, third line etc.
All the last few posts have done is create sone noise vaugely similar to whats required. This is possibly worse than going totally off post.


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off

Similar Threads
Thread Thread Starter Forum Replies Last Post
Loop through lines in file to find specified substring dresch Linux - Newbie 16 01-06-2007 05:34 PM
bash script loop bong.mau Programming 6 09-14-2005 07:38 PM
bash script for loop drisay Programming 5 12-25-2004 12:32 AM
Python - Set vars and loop over lines in file jnoller Programming 6 02-07-2004 10:32 AM
Loop over lines in text file? amaze Linux - General 2 08-12-2003 07:15 AM

All times are GMT -5. The time now is 05:04 PM.

Main Menu
Write for LQ is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration