Is there a record width limit in Linux
 

Hi Everyone,

I was wondering, is there a maximum record width for a file in Linux? I have plans for a very large file that is a little over 200,000 bytes wide and over 43,000 bytes long. Could Linux store a file such as this? I'm not worried about the lenght since I know that is a non-issue, I'm only worried about the width.

Thanks

I've just created a file with 100 records of size 200000 (just asked Python to write 100 times the same string and got a file of 20Mb). Nothing went wrong. The only thing I can imagine is that 200,000 x 43,000 is about 8 Gb, so you have to be sure that your kernel and filesystem can manage files of that size (which, I think, is the case for kernels from 2.4)

HTH

Thanks for the information. I will check the kernal out.

A file that is really that large and that wide, and not "sparse?" A file where you are not storing "a bunch of zeroes?" Rare, but possible.

Could the file not simply be stored as multiple, numbered files, in one directory? Upon reaching the end of one file, the application could search for the next one, switch to it, and return the "next record" from that file.

Well, for instance, some radiotelescopes generate a LOT of information (actually, so fast, that you need an army of harddisks to save it, even after a filter)

Okay... it still might be more manageable if the process wrote a certain amount of data to one file, then switched to the next file, periodically. A single file that is of exceptionally large size is rather ponderous to handle.

There's no such thing as a "record" in Linux, per se. Like Unix which inspired it, all files in Linux are made up of bytes, not records. As far as the OS and the file system is concerned, any file is just a string of bytes.

That's why everyone here is talking only in terms of file size limitations (in bytes).

It's entirely up to the software application to organize this string of bytes in terms of records within the file. Depending on what specific applications you're using to manipulate the file, there may be record width limitations.

If you use either Reiser or XFS, file size limitations will be pretty much limited by the size of your hard disk. EXT used to have a 2GB limit but I don't know what the limitation is.

Now, for "record size" I presume you're referring to a database, not flat files - which would depend on whether you want to run posgresql, MySQL, or other. Did you have a particular DBMS in mind?