LinuxQuestions.org

LinuxQuestions.org (/questions/)
-   Programming (http://www.linuxquestions.org/questions/programming-9/)
-   -   Read large text files (~10GB), parse for columns, output (http://www.linuxquestions.org/questions/programming-9/read-large-text-files-%7E10gb-parse-for-columns-output-717217/)

vache 04-06-2009 12:17 PM

Read large text files (~10GB), parse for columns, output
 
Hello, world.

The Goal: Read in ASCII text files, parse out specific columns, send to standard out.

I'm currently using a simple awk {print $1, ...} script to accomplish this. It's all fine and good, but the files I'm reading in are massive (10GB is not uncommon in our environment) and I speculate (hope) that a C or C++ application can parse these files faster than awk.

My C-fu is weak at best (featured below is my "101" level C code mushed together after lots of Google searches - ha) and it's actually slower than awk. If it matters, I have access to some very powerful hardware (64 bit quad core Xeon, 6GB of RAM).

What alternatives are there to fopen/etc. for reading in large files and parsing? Thanks in advance.

Code:

#include <stdio.h>
#include <string.h>

int main( int argc, char *argv[] )
{
    /* No file supplied? */
    if ( argc == 1 )
    {
        puts( "\nYou must supply a file to parse\n" );
        return 1;
    }

    /* Open the file, read-only */
    FILE *file = fopen( argv[1], "r" );

    /* If the file exists... */
    if ( file != NULL )
    {
        char line[256];
        char del[] = " ";

        /* While we can read a line from the file... */
        while ( fgets( line, sizeof line, file ) != NULL )
        {
            /* Convert each line in to tokens */
            char *result = NULL;
            result = strtok( line, del );
            int tkn = 1;

            /* "Foreach" token... */
            while( result != NULL )
            {
                /* If tkn matches our list, then print */
                /* $1, $2, $4, $6, $11, $12, $13 */
                if  (
                        tkn == 1 || tkn == 2 || tkn == 3 ||
                        tkn == 4 || tkn == 6 || tkn == 11 ||
                        tkn == 12 || tkn == 13
                    )
                {
                    printf( "%s ", result );
                }
                tkn++;
                result = strtok( NULL, del );
            }
        }
        fclose( file );
    } else {
        printf( "%s", argv[1] );
    }
    return 0;
}


Telemachos 04-06-2009 12:40 PM

What in the world could you do to a text file to make it 10GB? Wow.

Maybe I'm missing something, but if the main issue is simply that you can't load the whole file into memory at once, any solution would work if it read line by line. There are good, straightforward ways to do this in many scripting languages (Perl or Python, for example), and a higher level language would allow you to leverage very powerful built-in string techniques. I guess what I'm saying is that I don't know if C would offer a significant speed increase. The main speed issue is just going through all the lines, it seems, rather than any lower level algorithm. I'll be curious to hear if others know better.

Sergei Steshenko 04-06-2009 12:48 PM

Quote:

Originally Posted by vache (Post 3500207)
Hello, world.

The Goal: Read in ASCII text files, parse out specific columns, send to standard out.

I'm currently using a simple awk {print $1, ...} script to accomplish this. It's all fine and good, but the files I'm reading in are massive (10GB is not uncommon in our environment) and I speculate (hope) that a C or C++ application can parse these files faster than awk.

My C-fu is weak at best (featured below is my "101" level C code mushed together after lots of Google searches - ha) and it's actually slower than awk. If it matters, I have access to some very powerful hardware (64 bit quad core Xeon, 6GB of RAM).

What alternatives are there to fopen/etc. for reading in large files and parsing? Thanks in advance.

Code:

#include <stdio.h>
#include <string.h>

int main( int argc, char *argv[] )
{
    /* No file supplied? */
    if ( argc == 1 )
    {
        puts( "\nYou must supply a file to parse\n" );
        return 1;
    }

    /* Open the file, read-only */
    FILE *file = fopen( argv[1], "r" );

    /* If the file exists... */
    if ( file != NULL )
    {
        char line[256];
        char del[] = " ";

        /* While we can read a line from the file... */
        while ( fgets( line, sizeof line, file ) != NULL )
        {
            /* Convert each line in to tokens */
            char *result = NULL;
            result = strtok( line, del );
            int tkn = 1;

            /* "Foreach" token... */
            while( result != NULL )
            {
                /* If tkn matches our list, then print */
                /* $1, $2, $4, $6, $11, $12, $13 */
                if  (
                        tkn == 1 || tkn == 2 || tkn == 3 ||
                        tkn == 4 || tkn == 6 || tkn == 11 ||
                        tkn == 12 || tkn == 13
                    )
                {
                    printf( "%s ", result );
                }
                tkn++;
                result = strtok( NULL, del );
            }
        }
        fclose( file );
    } else {
        printf( "%s", argv[1] );
    }
    return 0;
}


I don't think you grounds to believe that your "C" code will be faster than awk or Perl.

vache 04-06-2009 01:50 PM

Quote:

Originally Posted by Telemachos (Post 3500238)
What in the world could you do to a text file to make it 10GB? Wow.

Infrastructure hardware on a class A network :)

vache 04-06-2009 01:51 PM

Quote:

Originally Posted by Sergei Steshenko (Post 3500248)
I don't think you grounds to believe that your "C" code will be faster than awk or Perl.

Hmm?

Sergei Steshenko 04-06-2009 01:58 PM

Quote:

Originally Posted by vache (Post 3500312)
Hmm?

Because, for example, Perl uses highly optimized regular expressions engine, and at all is highly optimized for text parsing.

Telemachos 04-06-2009 02:32 PM

@ Vache: Think about what you're doing here:
  • Open a file.
  • Start a loop which takes one line at a time from the file, saves state and ends when you hit EOF.
  • Check each line inside the loop for a match against a number of expressions.
  • Print the line if you hit a match and move onto the next line in the file. (I assume you want to print when you hit the first match and then skip the rest of the tests. No reason to keep testing the same line after you've hit a match.)
What I think Sergei is saying, and I am certainly saying, is that you don't have any special reason to think C will be significantly faster than Perl at doing those things. In addition, if developer time matters to you, then Perl is potentially faster (to develop) since it's built to handle strings, lines and regular expressions. Edit: That said, maybe I'm missing something obvious. I do that all the damn time.

jglands 04-06-2009 02:56 PM

Why not use VB?

Sergei Steshenko 04-06-2009 03:18 PM

Quote:

Originally Posted by jglands (Post 3500366)
Why not use VB?

The OP mentions awk, so he's most likely on UNIX-like system, and VB is unavailable.

jglands 04-06-2009 03:23 PM

He should install windows then.

Sergei Steshenko 04-06-2009 06:42 PM

Quote:

Originally Posted by jglands (Post 3500387)
He should install windows then.

What for ? And why to pay money for an OS which is definitely not necessary for the task ?

syg00 04-06-2009 07:24 PM

I/O is your problem - plain and simple. No matter how fast your CPU is, they all wait (for I/O completion in this case) at the same speed.

Go parse the first Gig of the data (only) - then go do it again. See the difference; that's caching versus real I/O.

Sergei Steshenko 04-07-2009 04:02 AM

Quote:

Originally Posted by syg00 (Post 3500582)
I/O is your problem - plain and simple. No matter how fast your CPU is, they all wait (for I/O completion in this case) at the same speed.

Go parse the first Gig of the data (only) - then go do it again. See the difference; that's caching versus real I/O.

That's right, the OS does wonders using smart caching, but for big files one can't fool nature.

jglands 04-07-2009 08:37 AM

He should use windows, because you get what you pay for. If it's free it must be junk.

Telemachos 04-07-2009 09:39 AM

Before anyone get's all riled up: please don't feed the troll.


All times are GMT -5. The time now is 01:48 AM.