Read large text files (~10GB), parse for columns, output

Sergei Steshenko · 04-07-2009, 09:51 AM

Quote:

Originally Posted by sundialsvcs

Stick to the subject, please... "Cheap beer and forums do not mix."

No, it probably won't be "better than awk."

"awk" is a very well-written program that is specialized for doing what you are doing.

All of the delays associated with this task will be mechanical ones: disk I/O times and network time. But "awk" knows to tell the operating-system that the file is being read sequentially, and therefore the operating system will know how to line-up lots of file buffers and other tricks to streamline the operation as much as the hardware will allow.

If the time required to do this task is problematic to the business, then there are various things that you can do:

Invest in fast storage-hardware... SATA, FireWire.
Instead of using the disk controllers built into the motherboard, buy a controller card. An inexpensive unit can make a dramatic difference.
Put the input file and the output file on different disk volumes.
Do not follow the siren that says, "put it all in memory..." Abandon all hope, ye who enter there!

Face it: when you're dealing with 10 gigabytes of data, "some things take time." If you're doing the task in "awk," and doing it well, then you are using a robust tool that was specifically designed for the task. You have not erred in the approach that you are using right now. "Diddling with it" will not improve it.

By the way, if I understood the OP correctly, the lines are independent, i.e. line by line parsing should be OK.

If it's the case, then the very first legitimate question is: "Why is it single 10GB file and not a number of much smaller files ?".

The point is that a number of files may be stored on separate hard drives and better yet the drives can be connected to different CPUs, so the whole processing can be done in parallel and the the results can be merged.

int0x80 · 04-07-2009, 09:52 AM

Quote:

Originally Posted by jglands

Well he would at least have support? What does he have from linux now? Some pimple faced kids telling him he is wrong instead of helping him.

I don't see any .NET devs on here showing him the way???

FAIL

jglands · 04-07-2009, 09:53 AM

That's right. He asked here instead of someplace that will help him.

int0x80 · 04-07-2009, 09:55 AM

Quote:

Originally Posted by jglands

That's right. He asked here instead of someplace that will help him.

Just because some of the members on here (Telemachos, Sergei) can't read or aren't smart enough to solve problems before posting doesn't mean that the entire community is worthless. LQ is representative of the internet with people of varying levels of intelligence. Some are stars (sundialsvcs), and others have no light on upstairs (jglands).

jglands · 04-07-2009, 09:59 AM

Quote:

Originally Posted by int0x80

Some are stars (sundialsvcs), and others have no light on upstairs (jglands).

Just because I have no hair doesn't mean my lights are not on. I could look like this guy.

int0x80 · 04-07-2009, 10:01 AM

Quote:

Originally Posted by jglands

Just because I have no hair doesn't mean my lights are not on. I could look like this guy.

Unfortunately you look like this...

Telemachos · 04-07-2009, 10:01 AM

Quote:

Originally Posted by int0x80

Just because some of the members on here (Telemachos, Sergei) can't read or aren't smart enough to solve problems before posting doesn't mean that the entire community is worthless.

Charming. Sergei and I said essentially the same thing as Sundialscvs, though I admit he said it more fully. What we all said was that the OP's C code was unlikely to beat a pre-existing tool (awk, Perl, Python, whatever) because the big issue was the simple math of the filesize.

jglands · 04-07-2009, 10:02 AM

Ok Code Monkey

vache · 04-07-2009, 10:03 AM

So, the short answer to my question is "no". Thanks

int0x80 · 04-07-2009, 10:06 AM

Quote:

Originally Posted by jglands

Ok Code Monkey

Clearly that is an image of your kin. Notice the `language name="C#"` in your image.

FAIL

Maybe one day the MS crowd will evolve to Linux.

jglands · 04-07-2009, 10:06 AM

See the guy has given up on Linux. About time.

int0x80 · 04-07-2009, 10:08 AM

The solution was not "use VB". You lose.

jglands · 04-07-2009, 10:08 AM

You just wish your stuff could be as good as C#. Good luck with finding your answer Vache. You won't get an answer from these preteens.

Sergei Steshenko · 04-07-2009, 10:09 AM

Quote:

Originally Posted by Telemachos

Charming. Sergei and I said essentially the same thing as Sundialscvs, though I admit he said it more fully. What we all said was that the OP's C code was unlikely to beat a pre-existing tool (awk, Perl, Python, whatever) because the big issue was the simple math of the filesize.

I once incidentally looked into Perl regular expressions code, which is a derived work of some standard RE library.

The most frequent comment was "we are doing/have changed this and that for efficiency reasons".

int0x80 · 04-07-2009, 10:10 AM

You wish C# could be as good as Java. Good luck with your Xtra Proprietary OS.