Maximum file size

Hiran Joseph · 11-22-2006, 01:56 AM

Does anyone know what is the maximum file size that can
be parsed by the AWK / or by linux in general?

bigearsbilly · 11-22-2006, 04:58 AM

linux file sizes, pretty large about 1TB i believe,
awk, dependent on memory and awk itself, dunno

with perl i have parsed very large files

jim mcnamara · 11-22-2006, 05:38 AM

Unless you mean "line size" -- which is usually thought of as the number of bytes per record in a text-formatted file -- if the filesystem can store the file, then utilities like awk and sed can read it.

What error are you getting?

matthewg42 · 11-22-2006, 05:39 AM

I once ran into a 2 GiB limit with awk on a HP-UX machine. I think it depends on which c library it was compiled against. You can probably tell by looking for the 64 bit file IO functions in the binary:

Code:

strings $(which awk) |grep fopen

If you see fopen64, I think you should be OK up to files of stupidly large size. If you see regular fopen, probably you'll hit the limit at 2 GiB.

randyding · 11-22-2006, 08:34 PM

sed also counts line numbers, there could be an issue if there is more than 2^32 lines streamed in or read from a file. I don't know if the following works, to print the 2^32 line number?? Anyone have a file this big laying around?

Code: