Share your knowledge at the LQ Wiki.
Go Back > Forums > Non-*NIX Forums > Programming
User Name
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.


  Search this Thread
Old 11-22-2006, 01:56 AM   #1
Hiran Joseph
LQ Newbie
Registered: Nov 2006
Posts: 1

Rep: Reputation: 0
Question Maximum file size

Does anyone know what is the maximum file size that can
be parsed by the AWK / or by linux in general?
Old 11-22-2006, 04:58 AM   #2
Senior Member
Registered: Mar 2004
Location: england
Distribution: NetBSD, Void, Debian, Mint, Ubuntu, Puppy, Raspbian
Posts: 3,487

Rep: Reputation: 233Reputation: 233Reputation: 233
linux file sizes, pretty large about 1TB i believe,
awk, dependent on memory and awk itself, dunno

with perl i have parsed very large files
Old 11-22-2006, 05:38 AM   #3
jim mcnamara
Registered: May 2002
Posts: 964

Rep: Reputation: 36
Unless you mean "line size" -- which is usually thought of as the number of bytes per record in a text-formatted file -- if the filesystem can store the file, then utilities like awk and sed can read it.

What error are you getting?
Old 11-22-2006, 05:39 AM   #4
Senior Member
Registered: Oct 2003
Location: UK
Distribution: Kubuntu 12.10 (using awesome wm though)
Posts: 3,530

Rep: Reputation: 65
I once ran into a 2 GiB limit with awk on a HP-UX machine. I think it depends on which c library it was compiled against. You can probably tell by looking for the 64 bit file IO functions in the binary:
strings $(which awk) |grep fopen
If you see fopen64, I think you should be OK up to files of stupidly large size. If you see regular fopen, probably you'll hit the limit at 2 GiB.
Old 11-22-2006, 08:34 PM   #5
Registered: May 2004
Posts: 552

Rep: Reputation: 31
sed also counts line numbers, there could be an issue if there is more than 2^32 lines streamed in or read from a file. I don't know if the following works, to print the 2^32 line number?? Anyone have a file this big laying around?
sed -n 4294967296p
Old 11-22-2006, 10:28 PM   #6
Senior Member
Registered: Jul 2005
Distribution: Slackware
Posts: 2,012

Rep: Reputation: 115Reputation: 115
The most general answer is as many bytes as you can index with an off_t.
Old 11-23-2006, 12:16 AM   #7
Registered: Jun 2005
Posts: 542

Rep: Reputation: 34
Originally Posted by tuxdev
The most general answer is as many bytes as you can index with an off_t.
Yes, it's the most reliable way. This and off64_t. As a matter of fact, the *BSD's lack fopen64() because they use a 64-bits wide off_t.
Old 11-24-2006, 07:54 AM   #8
Registered: Dec 2005
Distribution: RHEL3, FC3
Posts: 383

Rep: Reputation: 30
And fopen64 is a non-standard function, to support 64-bit file pointers


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off

Similar Threads
Thread Thread Starter Forum Replies Last Post
maximum linux file size? alec77 Linux - Newbie 8 12-03-2004 09:31 AM
avisync maximum file size david.skinner Linux - Software 0 11-26-2004 05:19 PM
Maximum file size with AIX & JFS ? stephnane AIX 4 07-07-2004 02:23 PM
Maximum Size of File in Red Hat Linux 2.1 imsajjadali Red Hat 0 07-02-2004 09:05 AM
dvd+rw maximum file size problem (burning) swamp2003 Linux - Software 1 12-10-2003 09:30 AM > Forums > Non-*NIX Forums > Programming

All times are GMT -5. The time now is 08:59 AM.

Main Menu
Write for LQ is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration