Large file size support for linux

merlin23 · 12-21-2004, 02:50 AM

I have a problem with my htdig-database-file...the file eats up more than 2 Gigs harddisk, which is the natural file size limit on standard-linux distros.

Now I heard about LFS(Large file size support) for linux...
Is it easy to enable this feature?
Do I have just to recompile my kernel with LFS-support?
Or isn't that so easy at all?

Otherwise I have to reduce the search-capability of my search enginge, which really isn't the finest way...

MasterC · 12-21-2004, 02:54 AM

The filesystem determines the filesize limit, not the OS. XFS, JFS, EXT3 all allow for files larger than 2GB. I believe it was just a limitation of ext2... I personally haven't had to 'enable' anything (maybe I have in the kernel and simply don't remember as it is just a matter of fact...) since ext3 came about.

HTH

Cool

merlin23 · 12-21-2004, 04:04 AM

if this would be the only problem, that I use ext2, I'd be really happy...

It's just because I've read the following tutorial about LFS from suse:

http://www.suse.de/~aj/linux_lfs.html

excerpt:

Quote:

LFS raises the limit of maximal file size. For 32-bit systems the limit is 231 (2 GiB) but using the LFS interface on filesystems that support LFS applications can handle files as large as 263 bytes.

For 64-bit systems the file size limit is 263 bytes unless a filesystem (like NFSv2) only supports less.

this seems to say, that it isn't the filesystem, but the architecture that it is depending on...

further more, there is following written:

Quote:

For using LFS in user programs, the programs have to use the LFS API. This involves recompilation and changes of programs. The API is documented in the glibc manual (the libc info pages) which can be read with e.g. "info libc".

In a nutshell for using LFS you can choose either of the following:

* Compile your programs with "gcc -D_FILE_OFFSET_BITS=64". This
forces all file access calls to use the 64 bit variants. Several types change
also, e.g. off_t becomes off64_t. It's therefore important to always use the
correct types and to not use e.g. int instead of off_t. For portability with
other platforms you should use getconf LFS_CFLAGS which will return
-D_FILE_OFFSET_BITS=64 on Linux platforms but might return something
else on e.g. Solaris. For linking, you should use the link flags that are
reported via getconf LFS_LDFLAGS. On Linux systems, you do not need
special link flags.

* Define _LARGEFILE_SOURCE and _LARGEFILE64_SOURCE. With
these defines you can use the LFS functions like open64 directly.

* Use the O_LARGEFILE flag with open to operate on large files.

thats what me made thinking about recompiling the kernel or the programs...

MasterC · 12-21-2004, 04:55 AM

Off the top of my head, I cannot recall where I recently saw those types of things being dealt with, however, LFS was needing to be enabled to do... something. But from my experience, I have had great success in using several applications (a common example is video editing) that work with large files on my HD. I generally use ext3 but have recently switched to XFS on some of my newer systems to check out it's 'performance enhancement' but either way, since I have switched up to ext3 I have not had any problems with large files (bigger than 2GB).

Cool

merlin23 · 12-21-2004, 05:06 AM

so the question is for me now, is changing to ext3 solving my problem??

Maybe I should give a try to recompile this mandrake-kernel...
allthough it seems a little bit difficult for me...

MasterC · 12-21-2004, 06:07 AM

To switch from ext2 to ext3, it's a tune2fs command:
tune2fs -j /dev/hdx

Where hdx is your device. However, I've been fortunate enough to be able to start with a fresh mke2fs -j so YMMV. However, it's certainly worth a shot, remember to remount the devices with ext3 rather than ext2 as your filesystem option:
mount -t ext3
OR
/etc/fstab entry change to ext3

HTH

If that is too vague, feel free to reply, I can go into more depth if you need it.

Cool

merlin23 · 12-21-2004, 06:20 AM

If you think it will help solving my problem I'll give it a try...

Are you sure that I don't loose my data with

tune2fs -j /dev/hda

?

It's my root partition...wouldn't be nice to loose it

MasterC · 12-22-2004, 02:32 AM

No it certainly wouldn't. No you are fine, if you run into problems, you can always revert back to mounting the device as ext2. You lose nothing but a bit of space for the journal.

Cool

merlin23 · 12-23-2004, 03:33 AM

I did this tune - comman dnow and there wasn't any errormessage...

Did it worked?
Is there a command to find out on which filesystem I'm running now?

MasterC · 12-23-2004, 03:56 AM

mount

Just plain with no delimiters. This will show you where everything is mounted and with what filesystem.

You will have to remount the device(s) with the ext3 option, it will not default to this. You will also have to edit fstab to have them mounted with the ext3 option upon successive reboots.

Cool

merlin23 · 01-10-2005, 08:10 AM

Hi anybody, I figured something out now:

It's not that my filesystem does not support >2GB file size, because I can make files larger than this actually!
(tried it with "dd if/dev/zero of=/test bs=1M count=3500")

Could it be that the app uses the fseek command, that handles ints, and not longs so there is only 32-bit support???
(I think I've found an issue about htat somewhere)

This would be the only idea I have...

What makes me wonder is that htdig claims to have no size limit in the documentation...
maybe I have to recompile the program against another glibc library?
Does anybody know how to do this??

merlin23 · 01-11-2005, 08:01 AM

Please help me !!!!
I don't know what to do...searched the whole net in meanwhile....