LinuxQuestions.org
Support LQ: Use code LQ3 and save $3 on Domain Registration
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices



Reply
 
Search this Thread
Old 02-28-2013, 01:42 PM   #1
wilelucho
LQ Newbie
 
Registered: Jun 2012
Location: Brazil
Distribution: Ubuntu
Posts: 19

Rep: Reputation: Disabled
[Solved] awk in a loop


Hi everyone,

I'm attempting to generate separate files coming from a single file
The source file has 112 rows and more than 1000 columns
What I want is to generate a file with the first, third, second columns and for each separated file I want to add the 4th column, the following file will have only the 5th line, and so on.
So, I tried with the following code
Code:
for i in {4..1000}
do
cat arquivototalt.txt | awk -F " " '{print $1,$3,$2,$i}' > ./arquivo_${i}.4c
done
 
Old 02-28-2013, 04:39 PM   #2
colucix
Moderator
 
Registered: Sep 2003
Location: Bologna
Distribution: CentOS 6.5 OpenSuSE 12.3
Posts: 10,509

Rep: Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957
Your loop is a little bit redundant, since the source file is parsed 997 times! Instead, let awk do the work in a single pass, e.g.
Code:
awk '{for (i = 4; i <= NF; i++) print $1, $3, $2, $i > sprintf("arquivo_%04d.4c",i)}' source_file
Hope this helps.
 
Old 03-01-2013, 08:11 AM   #3
wilelucho
LQ Newbie
 
Registered: Jun 2012
Location: Brazil
Distribution: Ubuntu
Posts: 19

Original Poster
Rep: Reputation: Disabled
Solved

Thanks colucix, it worked like a charm

Luis
 
Old 03-01-2013, 08:26 AM   #4
colucix
Moderator
 
Registered: Sep 2003
Location: Bologna
Distribution: CentOS 6.5 OpenSuSE 12.3
Posts: 10,509

Rep: Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957
You're welcome!
 
Old 03-02-2013, 06:36 AM   #5
grail
Guru
 
Registered: Sep 2009
Location: Perth
Distribution: Manjaro
Posts: 7,692

Rep: Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987
Please use the tools to mark your thread solved.
 
Old 03-02-2013, 12:21 PM   #6
David the H.
Bash Guru
 
Registered: Jun 2004
Location: Osaka, Japan
Distribution: Debian sid + kde 3.5 & 4.4
Posts: 6,823

Rep: Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950
For the record, the main problem with the OP loop is that the $i shell variable does not expand in the awk expression. Since the variable is hard-quoted, it won't expand, and awk ends up trying to use the non-existent awk variable 'i'.

In cases like this, you generally have to import shell variables into awk variables with the -v option.


I also imagine that the -F setting is probably unnecessary, since awk splits columns on all whitespace by default anyway. Unless of course the file contains tabs or other whitespace characters that you have to explicitly exclude from being treated as delimiters.


BTW, Useless Use Of Cat.


Finally, I highly recommend zero-padding the numbers in your filenames. It makes later processing much easier when the shell can sort them automatically.

Fortunately awk doesn't appear to be bothered by leading zeros in column expansion, so you can do the zero padding in the brace expansion (bash v4+).

Code:
for i in {0004..1000}; do
    awk -v "col=$i" '{print $1,$3,$2,$col}' arquivototalt.txt > "./arquivo_$i.4c"
done
If you're using an older version of bash, or another shell, you can pad the numbers with printf instead.

Code:
for i in {4..1000}; do
    awk -v "col=$i" '{print $1,$3,$2,$col}' arquivototalt.txt > "./arquivo_$(printf '%04d' "$i").4c"
done
 
Old 03-02-2013, 02:25 PM   #7
colucix
Moderator
 
Registered: Sep 2003
Location: Bologna
Distribution: CentOS 6.5 OpenSuSE 12.3
Posts: 10,509

Rep: Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957
David, your explanations are fully comprehensive and well phrased, as always. However, I would point out that running a single awk command, instead of 997 iterations is far more efficient.
 
Old 03-03-2013, 12:56 AM   #8
David the H.
Bash Guru
 
Registered: Jun 2004
Location: Osaka, Japan
Distribution: Debian sid + kde 3.5 & 4.4
Posts: 6,823

Rep: Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950
Oh, absolutely. I was taking that as a given since you had already addressed it. awk is definitely the proper choice here.

I just wanted to take the opportunity to address the shell code version as well, as a lesson in proper scripting technique.

Sorry if I didn't make that clear enough.
 
Old 03-04-2013, 06:02 AM   #9
wilelucho
LQ Newbie
 
Registered: Jun 2012
Location: Brazil
Distribution: Ubuntu
Posts: 19

Original Poster
Rep: Reputation: Disabled
thanks

Thanks David the H. and colucix, this post turned unexpectedly a useful class about awk

Regards
 
  


Reply

Tags
awk, recursive


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED]Wierd AWK behavior / AWK not reading first line. Involar Linux - Newbie 9 11-28-2012 11:53 AM
How to use awk in a for loop? cliffyao Programming 2 10-27-2010 11:51 PM
[SOLVED] awk for-loop babaqga Linux - Software 4 09-02-2010 04:57 AM
[SOLVED] for loop with variables musonio Linux - General 4 11-06-2009 03:45 PM
awk in loop How to Nkunzis Linux - Newbie 3 12-10-2006 02:34 PM


All times are GMT -5. The time now is 11:43 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration