LinuxQuestions.org
Latest LQ Deal: Latest LQ Deals
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices


Reply
  Search this Thread
Old 07-15-2010, 04:49 AM   #1
ilukacevic
LQ Newbie
 
Registered: Jul 2010
Posts: 22

Rep: Reputation: 0
awk multiple column into single column


Dear all,

I'm new to using awk or similar commands and I hope someone will be able to help me.

I have a multicolumn datas, like

a1 b1 ... f1
a2 b2 ... f2
. . ... .
. . ... .
. . ... .
an bn ... fn

I would like to make a file with all these data in one column, like

a1
a2
.
.
.
an
b1
b2
.
.
.
bn
.
.
.
f1
f2
.
.
.
fn


Can it be done with awk or some other command?
Also, is it possible then do add another column in front of this one with numbers of the lines (for every previous column), like

1 a1
2 a2
. .
. .
. .
n an
1 b1
2 b2
. .
. .
. .
n bn
. .
. .
. .
1 f1
2 f2
. .
. .
. .
n fn


Thank you all in advance!

Igor Lukacevic
 
Old 07-15-2010, 05:35 AM   #2
stuart_cherrington
Member
 
Registered: Aug 2008
Location: Get back in your Cube Unix Boy!
Distribution: rh5, oel5, debian etch, solaris
Posts: 228

Rep: Reputation: 36
cat <filename> | awk '{print "1 "$1}' > <newfile>
cat <filename> | awk '{print "1 "$2}' >> <newfile>

repeat for the amount of columns.

You could automate this a little by putting it into a script which works out how many columns are in the file, then repeats the above line until the columns are finished.

Stuart.
 
Old 07-15-2010, 06:03 AM   #3
H_TeXMeX_H
LQ Guru
 
Registered: Oct 2005
Location: $RANDOM
Distribution: slackware64
Posts: 12,928
Blog Entries: 2

Rep: Reputation: 1301Reputation: 1301Reputation: 1301Reputation: 1301Reputation: 1301Reputation: 1301Reputation: 1301Reputation: 1301Reputation: 1301Reputation: 1301
You can also do this within awk, you could make nested for loops using the NR and NF variables to go through and print all the columns. See:
http://www.grymoire.com/Unix/Awk.html
 
Old 07-15-2010, 06:04 AM   #4
colucix
LQ Guru
 
Registered: Sep 2003
Location: Bologna
Distribution: CentOS 6.5 OpenSuSE 12.3
Posts: 10,509

Rep: Reputation: 1983Reputation: 1983Reputation: 1983Reputation: 1983Reputation: 1983Reputation: 1983Reputation: 1983Reputation: 1983Reputation: 1983Reputation: 1983Reputation: 1983
A little more tricky solution without awk:
Code:
eval cat $(seq -f "<(cut -d' ' -f%.0f file)" 1 N)
where you have to substitute the fields in red with the actual values: the file name and the number of columns, respectively. It uses multiple process substitutions to feed the cat command. It assumes that columns in the original file are separated by space.

Last edited by colucix; 07-15-2010 at 06:06 AM.
 
1 members found this post helpful.
Old 07-15-2010, 06:49 AM   #5
ilukacevic
LQ Newbie
 
Registered: Jul 2010
Posts: 22

Original Poster
Rep: Reputation: 0
To stuart_cherrington:

Thanks for the tip! It nicely brings them all into one column. But it adds only number 1 to each row, and not the line numbers 1,...,n (as I draw in my first post).
Can this be amended?


To colucix:
Thanks for the tip! But your command line just gives me a blank terminal window. Nothing else.


To H_TeXMeX_H:
Thanks for the tip! I have a little trouble applying NR and NF loops. I need more time to see if it works.


Thank you all again!

Igor
 
Old 07-15-2010, 06:53 AM   #6
grail
LQ Guru
 
Registered: Sep 2009
Location: Perth
Distribution: Manjaro
Posts: 9,996

Rep: Reputation: 3187Reputation: 3187Reputation: 3187Reputation: 3187Reputation: 3187Reputation: 3187Reputation: 3187Reputation: 3187Reputation: 3187Reputation: 3187Reputation: 3187
Well I will take the awk challenge
Code:
awk '{for(i=1;i<=NF;i++)if(arr[i] ~ /./)arr[i]=arr[i]"\n"$i;else arr[i]=$i}END{for(x=1;x<=length(arr);x++)printf("%s\n",arr[x])}' in_file
Edit: Although there are probably issues if n is very large??

Last edited by grail; 07-15-2010 at 06:55 AM.
 
Old 07-15-2010, 07:20 AM   #7
konsolebox
Senior Member
 
Registered: Oct 2005
Distribution: Gentoo, Slackware, LFS
Posts: 2,248
Blog Entries: 8

Rep: Reputation: 235Reputation: 235Reputation: 235
For me the solution really depends on the total number of lines or at least the average. Also the problem tastes like C though.
 
Old 07-15-2010, 08:17 AM   #8
colucix
LQ Guru
 
Registered: Sep 2003
Location: Bologna
Distribution: CentOS 6.5 OpenSuSE 12.3
Posts: 10,509

Rep: Reputation: 1983Reputation: 1983Reputation: 1983Reputation: 1983Reputation: 1983Reputation: 1983Reputation: 1983Reputation: 1983Reputation: 1983Reputation: 1983Reputation: 1983
Quote:
Originally Posted by ilukacevic View Post
To colucix:
Thanks for the tip! But your command line just gives me a blank terminal window. Nothing else.
Uh, I'm sorry. Maybe it depends on what actually is the field separator in the original file.
 
Old 07-15-2010, 09:29 AM   #9
grail
LQ Guru
 
Registered: Sep 2009
Location: Perth
Distribution: Manjaro
Posts: 9,996

Rep: Reputation: 3187Reputation: 3187Reputation: 3187Reputation: 3187Reputation: 3187Reputation: 3187Reputation: 3187Reputation: 3187Reputation: 3187Reputation: 3187Reputation: 3187
Worked just fine for me ... thanx for the lesson as always colucix
 
Old 07-16-2010, 03:03 AM   #10
ilukacevic
LQ Newbie
 
Registered: Jul 2010
Posts: 22

Original Poster
Rep: Reputation: 0
Quote:
Originally Posted by colucix View Post
Uh, I'm sorry. Maybe it depends on what actually is the field separator in the original file.
The field separator is just 2 spaces. How does that influence the result?

Anyway, I used a mix of solution fromstuart_cherrington and H_TeXMeX_H and it worked just as I want it. The only thing I still lack is how to automatize several command with one loop. I tried with for loop, but failed. Here are the shell scripts I have.


This one works (there are only 6 columns in the original file, so it's not a problem):
#!/bin/sh

cat trf2_5_band2eps.freq | awk '{print NR,$1}' > trf2_5_band2eps_edit3.freq
cat trf2_5_band2eps.freq | awk '{print NR,$2}' >> trf2_5_band2eps_edit3.freq
cat trf2_5_band2eps.freq | awk '{print NR,$3}' >> trf2_5_band2eps_edit3.freq
cat trf2_5_band2eps.freq | awk '{print NR,$4}' >> trf2_5_band2eps_edit3.freq
cat trf2_5_band2eps.freq | awk '{print NR,$5}' >> trf2_5_band2eps_edit3.freq
cat trf2_5_band2eps.freq | awk '{print NR,$6}' >> trf2_5_band2eps_edit3.freq

awk '{print 10,$1,$2}' trf2_5_band2eps_edit3.freq > trf2_5_band2eps_edit2.freq



This one doesn't work. I would appreciate any help with it:
#!/bin/sh

cat trf2_5_band2eps.freq | awk '{print NR,$1}' > trf2_5_band2eps_edit3.freq
cat trf2_5_band2eps.freq | awk '{for(i=2; i<=6; i++) print NR,$i}' >> trf2_5_band2eps_edit3.freq

awk '{print 10,$1,$2}' trf2_5_band2eps_edit3.freq > trf2_5_band2eps_edit2.freq



Thank you all again!

Igor
 
Old 07-16-2010, 04:58 AM   #11
grail
LQ Guru
 
Registered: Sep 2009
Location: Perth
Distribution: Manjaro
Posts: 9,996

Rep: Reputation: 3187Reputation: 3187Reputation: 3187Reputation: 3187Reputation: 3187Reputation: 3187Reputation: 3187Reputation: 3187Reputation: 3187Reputation: 3187Reputation: 3187
Quote:
Originally Posted by ilukacevic
The field separator is just 2 spaces. How does that influence the result?
Well of course this will affect the result seeing the delimeter being passed to cut is a single space!!

So did you bother to try my solution? It is number of columns independent.

Quote:
Originally Posted by ilukacevic
This one doesn't work. I would appreciate any help with it:
#!/bin/sh

cat trf2_5_band2eps.freq | awk '{print NR,$1}' > trf2_5_band2eps_edit3.freq
cat trf2_5_band2eps.freq | awk '{for(i=2; i<=6; i++) print NR,$i}' >> trf2_5_band2eps_edit3.freq

awk '{print 10,$1,$2}' trf2_5_band2eps_edit3.freq > trf2_5_band2eps_edit2.freq
Hardly surprising. awk reads a file line by line and patterns / actions are performed on each line
So your second line will print the line number (NR) 5 times along with the corresponding field ($i) from that line

Hence why I have stored the results into an array to be delivered at the END of the script.

I also find it curious why you use 'cat' to pass the contents of a file to a program that reads the contents of a file???
 
Old 07-16-2010, 07:06 AM   #12
ilukacevic
LQ Newbie
 
Registered: Jul 2010
Posts: 22

Original Poster
Rep: Reputation: 0
Quote:
Originally Posted by grail View Post
Well of course this will affect the result seeing the delimeter being passed to cut is a single space!!

So did you bother to try my solution? It is number of columns independent.
Yes, I did. It's good, but it doesn't print out the line numbers as the first column. I tried adding the NF in printf, but it doesn't work.


Quote:
Originally Posted by grail View Post
Hardly surprising. awk reads a file line by line and patterns / actions are performed on each line
So your second line will print the line number (NR) 5 times along with the corresponding field ($i) from that line
I understand the mistake...is there a way around it (using this script)? Maybe smth similar to yours?

Hence why I have stored the results into an array to be delivered at the END of the script.

Quote:
Originally Posted by grail View Post
I also find it curious why you use 'cat' to pass the contents of a file to a program that reads the contents of a file???
I just copied what stuart_cherrington gave me, not thinking about it. Are you saying that it is unnecessary?


Igor
 
Old 07-16-2010, 07:14 AM   #13
ilukacevic
LQ Newbie
 
Registered: Jul 2010
Posts: 22

Original Poster
Rep: Reputation: 0
Also, a new issue arrised in the meanwhile. My data are of the format 2.33456D-05 and so on. I the end, I need them multiplied by a constant factor (decimal number 219474.6306726). But when I do that, it seems that awk doesn't understand the format of my data, and gives back incorrect results.

I know that one can ask for a certain format with printf, but how can I tell awk that the input data are of that format, so that it can read them correctly?

thnx

Igor
 
Old 07-16-2010, 07:29 AM   #14
MTK358
LQ 5k Club
 
Registered: Sep 2009
Posts: 6,443
Blog Entries: 3

Rep: Reputation: 723Reputation: 723Reputation: 723Reputation: 723Reputation: 723Reputation: 723Reputation: 723
Quote:
Originally Posted by ilukacevic View Post
Also, a new issue arrised in the meanwhile. My data are of the format 2.33456D-05 and so on. I the end, I need them multiplied by a constant factor (decimal number 219474.6306726).
I don't understand.

Also, why not Perl?
 
Old 07-16-2010, 07:34 AM   #15
colucix
LQ Guru
 
Registered: Sep 2003
Location: Bologna
Distribution: CentOS 6.5 OpenSuSE 12.3
Posts: 10,509

Rep: Reputation: 1983Reputation: 1983Reputation: 1983Reputation: 1983Reputation: 1983Reputation: 1983Reputation: 1983Reputation: 1983Reputation: 1983Reputation: 1983Reputation: 1983
Awk works in double precision but it's not aware of the fortran D notation. You should first convert D to E. Following my previous suggestion (but it can be applied for all the others) you can try something like:
Code:
eval cat $(seq -f "<(awk '{sub(/D/,\"E\"); print $%.0f''*''219474.6306726}' file)" 1 6) | nl
Edit: a little more explicit:
Code:
for i in $(seq 1 6)
do
  awk '{sub(/D/,"E"); print $'$i'*219474.6306726}' file
done | nl

Last edited by colucix; 07-16-2010 at 07:41 AM.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Column statistic by awk ? cs24 Programming 7 01-15-2010 05:41 AM
Read text file column by column RVF16 Programming 11 05-31-2009 07:16 AM
Change column value with AWK RyudoBlaze Programming 2 04-25-2009 03:05 AM
Concatenate column 1 and column 2 of related lines cgcamal Programming 4 11-20-2008 10:43 AM
awk column printing schneidz Programming 7 09-29-2005 06:14 AM

LinuxQuestions.org > Forums > Non-*NIX Forums > Programming

All times are GMT -5. The time now is 01:28 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration