LinuxQuestions.org
Go Job Hunting at the LQ Job Marketplace
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices

Reply
 
Search this Thread
Old 11-22-2006, 01:56 AM   #1
Hiran Joseph
LQ Newbie
 
Registered: Nov 2006
Posts: 1

Rep: Reputation: 0
Question Maximum file size


Does anyone know what is the maximum file size that can
be parsed by the AWK / or by linux in general?
 
Old 11-22-2006, 04:58 AM   #2
bigearsbilly
Senior Member
 
Registered: Mar 2004
Location: england
Distribution: FreeBSD, Debian, Mint, Puppy
Posts: 3,288

Rep: Reputation: 174Reputation: 174
linux file sizes, pretty large about 1TB i believe,
awk, dependent on memory and awk itself, dunno

with perl i have parsed very large files
 
Old 11-22-2006, 05:38 AM   #3
jim mcnamara
Member
 
Registered: May 2002
Posts: 964

Rep: Reputation: 34
Unless you mean "line size" -- which is usually thought of as the number of bytes per record in a text-formatted file -- if the filesystem can store the file, then utilities like awk and sed can read it.

What error are you getting?
 
Old 11-22-2006, 05:39 AM   #4
matthewg42
Senior Member
 
Registered: Oct 2003
Location: UK
Distribution: Kubuntu 12.10 (using awesome wm though)
Posts: 3,530

Rep: Reputation: 63
I once ran into a 2 GiB limit with awk on a HP-UX machine. I think it depends on which c library it was compiled against. You can probably tell by looking for the 64 bit file IO functions in the binary:
Code:
strings $(which awk) |grep fopen
If you see fopen64, I think you should be OK up to files of stupidly large size. If you see regular fopen, probably you'll hit the limit at 2 GiB.
 
Old 11-22-2006, 08:34 PM   #5
randyding
Member
 
Registered: May 2004
Posts: 552

Rep: Reputation: 31
sed also counts line numbers, there could be an issue if there is more than 2^32 lines streamed in or read from a file. I don't know if the following works, to print the 2^32 line number?? Anyone have a file this big laying around?
Code:
sed -n 4294967296p
 
Old 11-22-2006, 10:28 PM   #6
tuxdev
Senior Member
 
Registered: Jul 2005
Distribution: Slackware
Posts: 2,014

Rep: Reputation: 115Reputation: 115
The most general answer is as many bytes as you can index with an off_t.
 
Old 11-23-2006, 12:16 AM   #7
primo
Member
 
Registered: Jun 2005
Posts: 542

Rep: Reputation: 34
Quote:
Originally Posted by tuxdev
The most general answer is as many bytes as you can index with an off_t.
Yes, it's the most reliable way. This and off64_t. As a matter of fact, the *BSD's lack fopen64() because they use a 64-bits wide off_t.
 
Old 11-24-2006, 07:54 AM   #8
kshkid
Member
 
Registered: Dec 2005
Distribution: RHEL3, FC3
Posts: 383

Rep: Reputation: 30
And fopen64 is a non-standard function, to support 64-bit file pointers
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
maximum linux file size? alec77 Linux - Newbie 8 12-03-2004 09:31 AM
avisync maximum file size david.skinner Linux - Software 0 11-26-2004 05:19 PM
Maximum file size with AIX & JFS ? stephnane AIX 4 07-07-2004 02:23 PM
Maximum Size of File in Red Hat Linux 2.1 imsajjadali Red Hat 0 07-02-2004 09:05 AM
dvd+rw maximum file size problem (burning) swamp2003 Linux - Software 1 12-10-2003 09:30 AM


All times are GMT -5. The time now is 05:01 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration