LinuxQuestions.org
Go Job Hunting at the LQ Job Marketplace
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices



Reply
 
Search this Thread
Old 08-16-2005, 08:04 AM   #1
Hink
LQ Newbie
 
Registered: Jul 2005
Location: Chicago Suburbs, USA
Distribution: Slackware 11
Posts: 7

Rep: Reputation: 0
Is there a record width limit in Linux


Hi Everyone,

I was wondering, is there a maximum record width for a file in Linux? I have plans for a very large file that is a little over 200,000 bytes wide and over 43,000 bytes long. Could Linux store a file such as this? I'm not worried about the lenght since I know that is a non-issue, I'm only worried about the width.

Thanks
 
Old 08-16-2005, 08:36 AM   #2
enemorales
Member
 
Registered: Jul 2004
Location: Santiago, Chile
Distribution: Ubuntu
Posts: 410

Rep: Reputation: 30
I've just created a file with 100 records of size 200000 (just asked Python to write 100 times the same string and got a file of 20Mb). Nothing went wrong. The only thing I can imagine is that 200,000 x 43,000 is about 8 Gb, so you have to be sure that your kernel and filesystem can manage files of that size (which, I think, is the case for kernels from 2.4)

HTH
 
Old 08-16-2005, 08:55 AM   #3
Hink
LQ Newbie
 
Registered: Jul 2005
Location: Chicago Suburbs, USA
Distribution: Slackware 11
Posts: 7

Original Poster
Rep: Reputation: 0
Thanks for the information. I will check the kernal out.
 
Old 08-16-2005, 10:18 AM   #4
sundialsvcs
Guru
 
Registered: Feb 2004
Location: SE Tennessee, USA
Distribution: Gentoo, LFS
Posts: 5,455

Rep: Reputation: 1172Reputation: 1172Reputation: 1172Reputation: 1172Reputation: 1172Reputation: 1172Reputation: 1172Reputation: 1172Reputation: 1172
A file that is really that large and that wide, and not "sparse?" A file where you are not storing "a bunch of zeroes?" Rare, but possible.

Could the file not simply be stored as multiple, numbered files, in one directory? Upon reaching the end of one file, the application could search for the next one, switch to it, and return the "next record" from that file.
 
Old 08-17-2005, 03:22 AM   #5
enemorales
Member
 
Registered: Jul 2004
Location: Santiago, Chile
Distribution: Ubuntu
Posts: 410

Rep: Reputation: 30
Well, for instance, some radiotelescopes generate a LOT of information (actually, so fast, that you need an army of harddisks to save it, even after a filter)
 
Old 08-17-2005, 03:16 PM   #6
sundialsvcs
Guru
 
Registered: Feb 2004
Location: SE Tennessee, USA
Distribution: Gentoo, LFS
Posts: 5,455

Rep: Reputation: 1172Reputation: 1172Reputation: 1172Reputation: 1172Reputation: 1172Reputation: 1172Reputation: 1172Reputation: 1172Reputation: 1172
Okay... it still might be more manageable if the process wrote a certain amount of data to one file, then switched to the next file, periodically. A single file that is of exceptionally large size is rather ponderous to handle.
 
Old 08-17-2005, 03:33 PM   #7
IsaacKuo
Senior Member
 
Registered: Apr 2004
Location: Baton Rouge, Louisiana, USA
Distribution: Debian 4.0 Etch
Posts: 1,349

Rep: Reputation: 49
There's no such thing as a "record" in Linux, per se. Like Unix which inspired it, all files in Linux are made up of bytes, not records. As far as the OS and the file system is concerned, any file is just a string of bytes.

That's why everyone here is talking only in terms of file size limitations (in bytes).

It's entirely up to the software application to organize this string of bytes in terms of records within the file. Depending on what specific applications you're using to manipulate the file, there may be record width limitations.
 
Old 08-17-2005, 06:43 PM   #8
KimVette
Senior Member
 
Registered: Dec 2004
Location: Lee, NH
Distribution: OpenSUSE, CentOS, RHEL
Posts: 1,794

Rep: Reputation: 46
If you use either Reiser or XFS, file size limitations will be pretty much limited by the size of your hard disk. EXT used to have a 2GB limit but I don't know what the limitation is.

Now, for "record size" I presume you're referring to a database, not flat files - which would depend on whether you want to run posgresql, MySQL, or other. Did you have a particular DBMS in mind?
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
how to write a program to record audio in linux using c stephenwalter Programming 3 07-09-2012 08:20 PM
Unable to record mic-in with SoundBlaster Live! while able to record other sources max76230 Linux - Newbie 2 03-14-2005 05:31 AM
How to record audio in linux AndreTapxure Linux - Software 1 08-10-2004 05:17 AM
Quota issue, hard limit doesn't limit users Gratz Linux - Software 2 09-16-2003 08:35 AM
Linux in vague won't record audio. Help. roesingape Linux - Software 2 06-13-2003 10:08 AM


All times are GMT -5. The time now is 07:34 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration