LinuxQuestions.org
LinuxAnswers - the LQ Linux tutorial section.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - General
User Name
Password
Linux - General This Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.

Notices

Reply
 
Search this Thread
Old 04-17-2009, 07:50 PM   #1
CrendKing
LQ Newbie
 
Registered: Apr 2009
Posts: 2

Rep: Reputation: 0
Make sort ignore comment lines


I have a file in the following format

Code:
# comment 1
# comment 2
0 0 BBAAC 16
0 16 AACAD 16
0 32 ACDAC 16
...
I want to sort(1) it by the third column as string, which can be easily done by

Code:
sort --key=3 my_file
However, after sort, the first two comment lines may not be still at the head of the file. Is there a way to tell sort(1) to ignore the comment lines?

I assume there is no such convenient way. Then what I did is to
1) grep the comment lines to my_comments
2) delete the comment lines from my_file
3) sort my_file
4) concatenate my_comments with my_file

Step 1 can be easily done by

Code:
head -n 20 my_file | grep \# > my_comments
assuming there is always no more than 20 comment lines.

Step 2 is the source of headache. There seems no way to truncate file at head. I need to grep or sed my_file to a temporary new file, and sort the temporary file instead in step 3, right? If my_file is large, say 1GB, this will exhibit high performance and storage overhead. Is there in-place way?

Step 4, how to do it quickly without commands like

Code:
cat my_comments temp_file > my_file
?

In one word, is there a way to do in-place sort by taking advantage of knowing that the comment lines are few and only appear at head?

Thank you very much!
 
Old 04-17-2009, 08:56 PM   #2
ghostdog74
Senior Member
 
Registered: Aug 2006
Posts: 2,695
Blog Entries: 5

Rep: Reputation: 241Reputation: 241Reputation: 241
Code:
grep -v "^#" file |sort ...
 
Old 04-17-2009, 09:38 PM   #3
PTrenholme
Senior Member
 
Registered: Dec 2004
Location: Olympia, WA, USA
Distribution: Fedora, (K)Ubuntu
Posts: 4,147

Rep: Reputation: 330Reputation: 330Reputation: 330Reputation: 330
Or, if you don't have other comments in the file:

grep /^#/ file >/tmp/head;grep -v /^#/ | sort ... >/tmp/body; cat /tmp/head /tmp/body;rm -f /tmp/head /tmp/body

Note: Untested code.
 
Old 04-17-2009, 10:44 PM   #4
CrendKing
LQ Newbie
 
Registered: Apr 2009
Posts: 2

Original Poster
Rep: Reputation: 0
Thanks, folks. After having some understanding about sort's algorithm, I know that directly calling sort and grep | sort has no performance difference.

I noticed that zsh has ability to concatenate files with redirection (using >>). Thus the script can be

Code:
grep "^#" my_file > my_sorted
grep -v "^#" my_file | sort >> my_sorted
and the sort check and be done by

Code:
if [[ -n $(grep -v "^#" my_file | sort --check 2>&1) ]]; then
   ...
fi
Perfect! Linux is really fascinating...

Last edited by CrendKing; 04-17-2009 at 11:30 PM.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Comment lines 1-9, 11-19, 21-29 using Sed kushalkoolwal Programming 8 02-03-2009 04:02 PM
Comment/uncomment lines? Romanus81 Programming 4 03-31-2008 08:25 PM
listing non-comment lines scbops Linux - Newbie 5 09-18-2007 08:55 PM
Is there a line limit with the sort utility? Trying to sort 130 million lines of text gruffy Linux - General 4 08-10-2006 08:40 PM
awk/gawk/sed - read lines from file1, comment out or delete matching lines in file2 rascal84 Linux - General 1 05-24-2006 09:19 AM


All times are GMT -5. The time now is 04:00 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration