LinuxQuestions.org

LinuxQuestions.org (/questions/)
-   Programming (https://www.linuxquestions.org/questions/programming-9/)
-   -   Read large text files (~10GB), parse for columns, output (https://www.linuxquestions.org/questions/programming-9/read-large-text-files-%7E10gb-parse-for-columns-output-717217/)

Sergei Steshenko 04-07-2009 10:51 AM

Quote:

Originally Posted by sundialsvcs (Post 3501222)
:rolleyes: Stick to the subject, please... "Cheap beer and forums do not mix."

No, it probably won't be "better than awk."

"awk" is a very well-written program that is specialized for doing what you are doing.

All of the delays associated with this task will be mechanical ones: disk I/O times and network time. But "awk" knows to tell the operating-system that the file is being read sequentially, and therefore the operating system will know how to line-up lots of file buffers and other tricks to streamline the operation as much as the hardware will allow.

If the time required to do this task is problematic to the business, then there are various things that you can do:
  1. Invest in fast storage-hardware... SATA, FireWire.
  2. Instead of using the disk controllers built into the motherboard, buy a controller card. An inexpensive unit can make a dramatic difference.
  3. Put the input file and the output file on different disk volumes.
  4. Do not follow the siren that says, "put it all in memory..." Abandon all hope, ye who enter there!
Face it: when you're dealing with 10 gigabytes of data, "some things take time." If you're doing the task in "awk," and doing it well, then you are using a robust tool that was specifically designed for the task. You have not erred in the approach that you are using right now. "Diddling with it" will not improve it.

By the way, if I understood the OP correctly, the lines are independent, i.e. line by line parsing should be OK.

If it's the case, then the very first legitimate question is: "Why is it single 10GB file and not a number of much smaller files ?".

The point is that a number of files may be stored on separate hard drives and better yet the drives can be connected to different CPUs, so the whole processing can be done in parallel and the the results can be merged.

int0x80 04-07-2009 10:52 AM

Quote:

Originally Posted by jglands (Post 3501233)
Well he would at least have support? What does he have from linux now? Some pimple faced kids telling him he is wrong instead of helping him.

I don't see any .NET devs on here showing him the way???

FAIL

jglands 04-07-2009 10:53 AM

That's right. He asked here instead of someplace that will help him.

int0x80 04-07-2009 10:55 AM

Quote:

Originally Posted by jglands (Post 3501240)
That's right. He asked here instead of someplace that will help him.

Just because some of the members on here (Telemachos, Sergei) can't read or aren't smart enough to solve problems before posting doesn't mean that the entire community is worthless. LQ is representative of the internet with people of varying levels of intelligence. Some are stars (sundialsvcs), and others have no light on upstairs (jglands).

jglands 04-07-2009 10:59 AM

1 Attachment(s)
Quote:

Originally Posted by int0x80 (Post 3501246)
Some are stars (sundialsvcs), and others have no light on upstairs (jglands).

Just because I have no hair doesn't mean my lights are not on. I could look like this guy.

int0x80 04-07-2009 11:01 AM

1 Attachment(s)
Quote:

Originally Posted by jglands (Post 3501251)
Just because I have no hair doesn't mean my lights are not on. I could look like this guy.

Unfortunately you look like this...

Telemachos 04-07-2009 11:01 AM

Quote:

Originally Posted by int0x80 (Post 3501246)
Just because some of the members on here (Telemachos, Sergei) can't read or aren't smart enough to solve problems before posting doesn't mean that the entire community is worthless.

Charming. Sergei and I said essentially the same thing as Sundialscvs, though I admit he said it more fully. What we all said was that the OP's C code was unlikely to beat a pre-existing tool (awk, Perl, Python, whatever) because the big issue was the simple math of the filesize.

jglands 04-07-2009 11:02 AM

1 Attachment(s)
Ok Code Monkey

vache 04-07-2009 11:03 AM

So, the short answer to my question is "no". Thanks :)

int0x80 04-07-2009 11:06 AM

Quote:

Originally Posted by jglands (Post 3501257)
Ok Code Monkey

Clearly that is an image of your kin. Notice the `language name="C#"` in your image.

FAIL

Maybe one day the MS crowd will evolve to Linux.

jglands 04-07-2009 11:06 AM

See the guy has given up on Linux. About time.

int0x80 04-07-2009 11:08 AM

The solution was not "use VB". You lose.

jglands 04-07-2009 11:08 AM

You just wish your stuff could be as good as C#. Good luck with finding your answer Vache. You won't get an answer from these preteens.

Sergei Steshenko 04-07-2009 11:09 AM

Quote:

Originally Posted by Telemachos (Post 3501254)
Charming. Sergei and I said essentially the same thing as Sundialscvs, though I admit he said it more fully. What we all said was that the OP's C code was unlikely to beat a pre-existing tool (awk, Perl, Python, whatever) because the big issue was the simple math of the filesize.

I once incidentally looked into Perl regular expressions code, which is a derived work of some standard RE library.

The most frequent comment was "we are doing/have changed this and that for efficiency reasons".

int0x80 04-07-2009 11:10 AM

You wish C# could be as good as Java. Good luck with your Xtra Proprietary OS.


All times are GMT -5. The time now is 03:53 AM.