LinuxQuestions.org
Review your favorite Linux distribution.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 08-12-2012, 03:00 PM   #1
sal_x_sal
LQ Newbie
 
Registered: Jan 2012
Posts: 7

Rep: Reputation: Disabled
Extracting second line from multiple txt files and append to a file


Hi,

I am trying to use an awk command to extract the second line from a bunch of txt files (tab delimited) and append them to a new file.

I use the command below to extract a second line from a single txt file, and it works fine.
awk 'NR==2' old_file1.txt > new_file1.txt

I know I have to have some kind of a loop to go over all the input files but I am not very much familiar with awk commands, any help will be much appreciated.

Thanks!
 
Old 08-12-2012, 03:21 PM   #2
Didier Spaier
LQ Addict
 
Registered: Nov 2008
Location: Paris, France
Distribution: Slint64-14.2beta2 on Lenovo Thinkpad W520
Posts: 6,780

Rep: Reputation: 2117Reputation: 2117Reputation: 2117Reputation: 2117Reputation: 2117Reputation: 2117Reputation: 2117Reputation: 2117Reputation: 2117Reputation: 2117Reputation: 2117
"man bash" + Bash Guide for Beginners and others at http://tldp.org
 
Old 08-12-2012, 03:30 PM   #3
i_joh
Member
 
Registered: Apr 2005
Distribution: Debian
Posts: 82

Rep: Reputation: 5
Code:
for FILE in *.txt; do sed -n '2p' $FILE; done > output
Just change *.txt to whatever. Could be a list of files separated by spaces too.
 
Old 08-12-2012, 03:57 PM   #4
sal_x_sal
LQ Newbie
 
Registered: Jan 2012
Posts: 7

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by i_joh View Post
Code:
for FILE in *.txt; do sed -n '2p' $FILE; done > output
Just change *.txt to whatever. Could be a list of files separated by spaces too.

Thanks for the response. I tried that and it worked perfectly except for one thing, the order is not maintained. May be I should ask this: how are the input files processed? Isnt it by file name? I wanted my output to be the same order as the inputfiles so that I can know which output line is coming from which file.
 
Old 08-12-2012, 04:10 PM   #5
i_joh
Member
 
Registered: Apr 2005
Distribution: Debian
Posts: 82

Rep: Reputation: 5
I get alphabetical sorting. What happens is that *.txt is replaced with all the .txt files in the current directory, each separated by a space. Then the first file name is put in $FILE and the command run. Then the second file name is put in $FILE and the command run, etc until all files are processed. An alternative would be to list the files manually in the order you want them listed, or sort them with the ls command (see man ls for options):

Code:
for FILE in $(ls -r); do sed -n '2p' "$FILE"; done
That would sort them in reverse.
 
Old 08-12-2012, 04:13 PM   #6
sal_x_sal
LQ Newbie
 
Registered: Jan 2012
Posts: 7

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by i_joh View Post
Code:
for FILE in *.txt; do sed -n '2p' $FILE; done > output
Just change *.txt to whatever. Could be a list of files separated by spaces too.
Or is there a way to include a column in the output file containing the name of the input files? So every time a second line is read from the input file, it is appended to the output file with an extra column added containing the file name. This would be equally usefull if it is impossible to guarantee that the input files will be processed in the same order. Thanks!
 
Old 08-12-2012, 04:16 PM   #7
i_joh
Member
 
Registered: Apr 2005
Distribution: Debian
Posts: 82

Rep: Reputation: 5
Quote:
Originally Posted by sal_x_sal View Post
Or is there a way to include a column in the output file containing the name of the input files? So every time a second line is read from the input file, it is appended to the output file with an extra column added containing the file name. This would be equally usefull if it is impossible to guarantee that the input files will be processed in the same order. Thanks!
Code:
for FILE in *.txt; do echo "$FILE: $(sed -n '2p' $FILE)"; done > output
 
Old 08-12-2012, 04:24 PM   #8
ntubski
Senior Member
 
Registered: Nov 2005
Distribution: Arch
Posts: 3,013

Rep: Reputation: 1225Reputation: 1225Reputation: 1225Reputation: 1225Reputation: 1225Reputation: 1225Reputation: 1225Reputation: 1225Reputation: 1225
Code:
awk 'FNR==2{print FILENAME, $0}' *.txt > output
 
Old 08-12-2012, 04:40 PM   #9
sal_x_sal
LQ Newbie
 
Registered: Jan 2012
Posts: 7

Original Poster
Rep: Reputation: Disabled
Thank you both for the responses. Both work perfect and produced the same result. One little thing if it is easier to do in awk, otherwise I will have to do the hard way. Basically, the filenames are now mixed together with the first column of the input file. Is there a way to have a tab after the file name, so the input file name is in a different column? Thank you very much once again!
 
Old 08-12-2012, 04:45 PM   #10
i_joh
Member
 
Registered: Apr 2005
Distribution: Debian
Posts: 82

Rep: Reputation: 5
Quote:
Originally Posted by sal_x_sal View Post
Thank you both for the responses. Both work perfect and produced the same result. One little thing if it is easier to do in awk, otherwise I will have to do the hard way. Basically, the filenames are now mixed together with the first column of the input file. Is there a way to have a tab after the file name, so the input file name is in a different column? Thank you very much once again!
Code:
for FILE in *.txt; do echo -e "$FILE\t$(sed -n '2p' $FILE)"; done > output
I'd recommend you learn shell programming. It's kind of what makes Linux/BSD so much better than Windows.
 
Old 08-12-2012, 04:48 PM   #11
ntubski
Senior Member
 
Registered: Nov 2005
Distribution: Arch
Posts: 3,013

Rep: Reputation: 1225Reputation: 1225Reputation: 1225Reputation: 1225Reputation: 1225Reputation: 1225Reputation: 1225Reputation: 1225Reputation: 1225
Code:
awk -vOFS='\t' 'FNR==2{print FILENAME, $0}' *.txt > output
 
Old 08-12-2012, 05:00 PM   #12
sal_x_sal
LQ Newbie
 
Registered: Jan 2012
Posts: 7

Original Poster
Rep: Reputation: Disabled
Both commands work perfect. Thank you very much i_joh and ntubski for your time in helping me with this. I appreciate it.

@i_joh - I am so much sucked working with windows in my daily life. Once in a while when I have to use linux I realize how powerful it is. I try to learn linux on my own but since I am not using it much I end up forgetting a lot of stuff. Thanks once again and have a good day!
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Writing multiple files from one file after comparing each line jlrod912 Linux - Newbie 2 06-26-2012 04:05 PM
[SOLVED] Read from Input File and Append to Multiple Files bridrod Linux - Newbie 8 11-24-2010 08:05 PM
Copy the contents of a txt file to other txt files (with similar names) by cp command TheIndependentAquarius Linux - Newbie 7 07-03-2010 01:54 AM
AWK/Perl for extracting data from txt file to numerous other files briana.paige Linux - Newbie 2 05-05-2009 10:53 AM
Create 1 csv file from multiple txt files richmur Programming 10 09-03-2008 02:28 PM


All times are GMT -5. The time now is 03:01 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration