does anyone know of a way to get a file line count for large files that doesn't use "wc -l" or "grep -c" etc.?

there are countless ways to do that.

Is this homework?

I'm just looking for a better way to calculate the line numbers from a file that builds to several hundred MB during the course of a day that takes seconds or less and doesn't require using an editor. has to be command line.

Hi,

Here's one way that seems to be faster then wc -l.

sed -n '$=' infile

I don't know if this is also true for large (a few hundred Mb or larger) files.

Here's one using perl, maybe not the fastest but it will be able to handle very large files:

perl -lne 'END { print $. }' < infile

Anyway, hope this helps.

*shrug*. This one seems to run a bit faster than the sed one in the previous post runs (at least, on my system). More complicated too.

nl infile | tac |sed -n 1p | awk '{print $1}

Hi,

Creative!

The execution speed depends on a lot of things and my result (or yours) may not be correct for others. Here are some anyway:

nl infile | tac |sed -n 1p | awk '{print $1} vs sed -n '$=' infile vs wc -l infile
infile: 500000 lines (half a million), 18 Mb flat text file.

5 runs each, only last 4 are counted.

0.024 sec (avg) - wc -l infile
0.121 sec (avg) - sed -n '$=' infile
0.396 sec (avg) - nl infile | tac |sed -n 1p | awk '{print $1}

On my box wc -l wins hands down. But that was to be expected; it was written especially for counting lines/words/bytes.

I disagree about the 'more complicated part'. How can one, small and simple, command be more complicated then 4 different, piped, commands

1 members found this post helpful.

Hi,

Just came to mind (although kinda obvious): awk '{ x++ } END { print x }' infile

It's a bit faster(0.099 sec) then the sed solution.

thanks for all the great ideas.

The sed solution offered by druuna just has to be the choice. sed always wins these simple problems - easy to do, easy to remember.
I suspect all the tests above will be invalid - unless specific steps were taken between every run to purge the disk cache.

Edit: still can't understand why the obvious (wc -l) isn't acceptable.

If this is a log file that grows over time, you could use dd with a block size of 1 and an offset equal to the size of the file the last time you ran your check. This will output the new contents of the file, which you could pipe through wc -l.
Record the size of the file each time you run the check.

For a file this large, maybe using the size of the file instead of the number of lines would be the information you should be using.

I looked for answers to this question and saw complicated versions 2 inches long. This one came to mind and I tried it on several files:

grep -c "" /home/usr1/my-file.txt

Will this always work?

Thanks to all.

@vouser: Why not use wc -l <file>? wc was written for this task and is by far the fastest.

BTW: Your grep solution does work.

If you just need the estimate of count and not the exact count, there is always something called "Mathemetics".

1. Do a head -1 FILE >NEW_FILE
2. get size of NEW_FILE
3. Get size of FILE
4. Divide result of 3 by result of 2 to get an estimated row count.

To get better result estimate, do a head -n and adjust accordingly during division.

I have almost 46 Gig file still growing in size, doing a wc -l or any other methods would take quite a bit. I do above to estimate when my process would finish (An oracle spool of 300 Mn Records). Just a thought....

1 members found this post helpful.

wc -l counts the number of newlines and as such has to scan to the end of each line. To speed it up you can first use cut to reduce the lines down to a couple characters and pipe it into wc. This can cut the time down to less than half of what it takes to just run wc.

eg. cut -c -2 hugefile.txt | wc -l