Linux - NewbieThis Linux forum is for members that are new to Linux.
Just starting out and have a question?
If it is not in the man pages or the how-to's this is the place!
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Introduction to Linux - A Hands on Guide
This guide was created as an overview of the Linux Operating System, geared toward new users as an exploration tour and getting started guide, with exercises at the end of each chapter.
For more advanced trainees it can be a desktop reference, and a collection of the base knowledge needed to proceed with system and network administration. This book contains many real life examples derived from the author's experience as a Linux system and network administrator, trainer and consultant. They hope these examples will help you to get a better understanding of the Linux system and that you feel encouraged to try out things on your own.
Click Here to receive this Complete Guide absolutely free.
etc -- all on same line, tab delimited. What I want to do is sort the file by the first segment (Strep_excerpt_1900_100_15_164_145/1). I am sorting because each entry as the one just posted, has a match pair that will have the same identifier but a 1 or 2 at the end, */[1,2] and I'd like them to go together. I was thinking of doing this in perl but the memory requirement would be ridiculous and it would run ever so slowly. Is there a quick sort command that will do something like this?
I was thinking of doing this in perl but the memory requirement would be ridiculous and it would run ever so slowly. Is there a quick sort command that will do something like this?
perl is interpreted as is bash and depending on how large your file is, bash would take a similar amount of memory as perl, possibly more.
For clarification; the entire file is all on one line? or just each entry starting with Strep and ending with /?. Your post would lead us to believe that there are multiple lines. Try using the QUOTE or CODE formatting buttons on the LQ post window
Actully, Perl is NOT interpreted as per bash. The actual perl prog (not the one you write) reads your prog and basically compiles it on the fly and runs the 'compiled' version.
For the gory details : http://www.perl.com/doc/FMTEYEWTK/comp-vs-interp.html
The net effect is about 80-90% speed of C.
OP qn: I agree, its not clear where the end of record is.
Does each rec start with 'Strep_excerp..' and end with a newline, or have you got multiple 'logical' ( Strep_excerp) recs on each physical line?
The other cols are tab separated I understand.
As mentioned, please the CODE tags option.
Using just sort on millions of lines may cause an out of memory error, or caching to the swap partition.
What does the number before the /1 or /2 signify, and do they appear in random locations. You may be able to exploit the fact that there are two of them in some way. Another possibility is splitting up the original file into separate files based on the number before the slash and then sorting the temporary slices before cat'ing them together.
Looking at your first example, the entries look presorted. If all /1 lines are presorted and all /2 lines are presorted, but the two types of lines together aren't, you could filter all /1 lines in one stream, all /2 lines in another stream and then merge the streams together with "sort -m". I don't think you would have the same memory problem. Each of the two inputs needs to be presorted, so the program doesn't need to hold all of the lines in memory. As soon as a line from both streams is greater then the previous line read in, the previous line can be printed out and the memory reclaimed.