LinuxQuestions.org
Latest LQ Deal: Linux Power User Bundle
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 08-24-2009, 07:55 AM   #1
turquoise_man
LQ Newbie
 
Registered: Aug 2009
Posts: 2

Rep: Reputation: 0
How to sort a csv on three fields ? (cygwin user)


Hi people

I have a large csv file (50 fields approx 1.4 million rows)

i want to sort the data on three fields in order say 5 then 3 then 6

I have done this with two fields, i.e

sort -t -k 5,5 -k 3,3 <filename>

but it doesn't seem to work for three i.e.

sort -t -k 5,5 -k 3,3 - k 6,6 <filename>

is this possible ?

I have written a perl script which does this correctly but runs into memory issues due to the file's size.

Would there be a way I could apply the perl algorithm (it works on an array of arrays) using the linux sort script (which has no problem with the memory)?

Any comments would be much appreciated.

(aplologiers in advance but I can't show the actual file due to privacy issues)
 
Old 08-24-2009, 08:34 AM   #2
pixellany
LQ Veteran
 
Registered: Nov 2005
Location: Annapolis, MD
Distribution: Arch/XFCE
Posts: 17,802

Rep: Reputation: 738Reputation: 738Reputation: 738Reputation: 738Reputation: 738Reputation: 738Reputation: 738
You are using the "-t" flag to designate a field separator, but then you don't specify anything. (If you have a CSV file, then maybe you need a ",".

Why the ranges? (6,6 3,3 ...etc.)
 
Old 08-24-2009, 09:12 AM   #3
turquoise_man
LQ Newbie
 
Registered: Aug 2009
Posts: 2

Original Poster
Rep: Reputation: 0
Quote:
Originally Posted by pixellany View Post
You are using the "-t" flag to designate a field separator, but then you don't specify anything. (If you have a CSV file, then maybe you need a ",".

Why the ranges? (6,6 3,3 ...etc.)
Sorry

it should read -t ","

5,5 as in the fifth column and end there, then the 3rd colum and end there e.t.c. This may not be doing quite what I want though..

I want to sort on the 5h colum then if equal sort on the 3rd en if these colums are equal sort on the 5h.

If it helps the perl routine for doing this is:

Code:
sub tableSorter($$){
	my ($row1,$row2) = @_;
	
	my $column6comparison = $row1->[5] cmp $row2->[5];
	if($column6omparison != 0) {
		return $column10comparison;
	}

	my $column3omparison = $row1->[3] cmp $row2->[3];
	if($column3comparison != 0) {
		return $column3comparison;
	}

	return $row1->[6] <=> $row2->[6];
	
}

Last edited by turquoise_man; 08-24-2009 at 11:35 AM. Reason: code tag wrong
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
awk question on handling *.CSV "text fields" in awk jschiwal Programming 8 05-27-2010 06:23 AM
user root does not exist (Cygwin) homer_3 Linux - General 1 07-30-2009 12:39 AM
Parsing a comma separated CSV file where fields have commas in to trickyflash Linux - General 7 03-26-2009 03:30 PM
Comparing two csv files and write different record in third CSV file irfanb146 Linux - Newbie 3 06-30-2008 09:15 PM
Unix sort on multiple fields belorion Programming 8 11-12-2007 01:15 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie

All times are GMT -5. The time now is 12:54 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration