LinuxQuestions.org
Visit Jeremy's Blog.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 12-21-2004, 03:50 AM   #1
merlin23
Member
 
Registered: Dec 2004
Location: Vienna
Posts: 46

Rep: Reputation: 15
Large file size support for linux


I have a problem with my htdig-database-file...the file eats up more than 2 Gigs harddisk, which is the natural file size limit on standard-linux distros.

Now I heard about LFS(Large file size support) for linux...
Is it easy to enable this feature?
Do I have just to recompile my kernel with LFS-support?
Or isn't that so easy at all?

Otherwise I have to reduce the search-capability of my search enginge, which really isn't the finest way...
 
Old 12-21-2004, 03:54 AM   #2
MasterC
LQ Guru
 
Registered: Mar 2002
Location: Salt Lake City, UT - USA
Distribution: Gentoo ; LFS ; Kubuntu
Posts: 12,612

Rep: Reputation: 64
The filesystem determines the filesize limit, not the OS. XFS, JFS, EXT3 all allow for files larger than 2GB. I believe it was just a limitation of ext2... I personally haven't had to 'enable' anything (maybe I have in the kernel and simply don't remember as it is just a matter of fact...) since ext3 came about.

HTH

Cool
 
Old 12-21-2004, 05:04 AM   #3
merlin23
Member
 
Registered: Dec 2004
Location: Vienna
Posts: 46

Original Poster
Rep: Reputation: 15
if this would be the only problem, that I use ext2, I'd be really happy...

It's just because I've read the following tutorial about LFS from suse:

http://www.suse.de/~aj/linux_lfs.html

excerpt:

Quote:
LFS raises the limit of maximal file size. For 32-bit systems the limit is 231 (2 GiB) but using the LFS interface on filesystems that support LFS applications can handle files as large as 263 bytes.

For 64-bit systems the file size limit is 263 bytes unless a filesystem (like NFSv2) only supports less.

this seems to say, that it isn't the filesystem, but the architecture that it is depending on...

further more, there is following written:



Quote:
For using LFS in user programs, the programs have to use the LFS API. This involves recompilation and changes of programs. The API is documented in the glibc manual (the libc info pages) which can be read with e.g. "info libc".

In a nutshell for using LFS you can choose either of the following:

* Compile your programs with "gcc -D_FILE_OFFSET_BITS=64". This
forces all file access calls to use the 64 bit variants. Several types change
also, e.g. off_t becomes off64_t. It's therefore important to always use the
correct types and to not use e.g. int instead of off_t. For portability with
other platforms you should use getconf LFS_CFLAGS which will return
-D_FILE_OFFSET_BITS=64 on Linux platforms but might return something
else on e.g. Solaris. For linking, you should use the link flags that are
reported via getconf LFS_LDFLAGS. On Linux systems, you do not need
special link flags.

* Define _LARGEFILE_SOURCE and _LARGEFILE64_SOURCE. With
these defines you can use the LFS functions like open64 directly.

* Use the O_LARGEFILE flag with open to operate on large files.


thats what me made thinking about recompiling the kernel or the programs...
 
Old 12-21-2004, 05:55 AM   #4
MasterC
LQ Guru
 
Registered: Mar 2002
Location: Salt Lake City, UT - USA
Distribution: Gentoo ; LFS ; Kubuntu
Posts: 12,612

Rep: Reputation: 64
Off the top of my head, I cannot recall where I recently saw those types of things being dealt with, however, LFS was needing to be enabled to do... something. But from my experience, I have had great success in using several applications (a common example is video editing) that work with large files on my HD. I generally use ext3 but have recently switched to XFS on some of my newer systems to check out it's 'performance enhancement' but either way, since I have switched up to ext3 I have not had any problems with large files (bigger than 2GB).

Cool
 
Old 12-21-2004, 06:06 AM   #5
merlin23
Member
 
Registered: Dec 2004
Location: Vienna
Posts: 46

Original Poster
Rep: Reputation: 15
so the question is for me now, is changing to ext3 solving my problem??

Maybe I should give a try to recompile this mandrake-kernel...
allthough it seems a little bit difficult for me...
 
Old 12-21-2004, 07:07 AM   #6
MasterC
LQ Guru
 
Registered: Mar 2002
Location: Salt Lake City, UT - USA
Distribution: Gentoo ; LFS ; Kubuntu
Posts: 12,612

Rep: Reputation: 64
To switch from ext2 to ext3, it's a tune2fs command:
tune2fs -j /dev/hdx

Where hdx is your device. However, I've been fortunate enough to be able to start with a fresh mke2fs -j so YMMV. However, it's certainly worth a shot, remember to remount the devices with ext3 rather than ext2 as your filesystem option:
mount -t ext3
OR
/etc/fstab entry change to ext3

HTH

If that is too vague, feel free to reply, I can go into more depth if you need it.

Cool
 
Old 12-21-2004, 07:20 AM   #7
merlin23
Member
 
Registered: Dec 2004
Location: Vienna
Posts: 46

Original Poster
Rep: Reputation: 15
If you think it will help solving my problem I'll give it a try...

Are you sure that I don't loose my data with

tune2fs -j /dev/hda

?

It's my root partition...wouldn't be nice to loose it
 
Old 12-22-2004, 03:32 AM   #8
MasterC
LQ Guru
 
Registered: Mar 2002
Location: Salt Lake City, UT - USA
Distribution: Gentoo ; LFS ; Kubuntu
Posts: 12,612

Rep: Reputation: 64
No it certainly wouldn't. No you are fine, if you run into problems, you can always revert back to mounting the device as ext2. You lose nothing but a bit of space for the journal.

Cool
 
Old 12-23-2004, 04:33 AM   #9
merlin23
Member
 
Registered: Dec 2004
Location: Vienna
Posts: 46

Original Poster
Rep: Reputation: 15
I did this tune - comman dnow and there wasn't any errormessage...

Did it worked?
Is there a command to find out on which filesystem I'm running now?
 
Old 12-23-2004, 04:56 AM   #10
MasterC
LQ Guru
 
Registered: Mar 2002
Location: Salt Lake City, UT - USA
Distribution: Gentoo ; LFS ; Kubuntu
Posts: 12,612

Rep: Reputation: 64
mount

Just plain with no delimiters. This will show you where everything is mounted and with what filesystem.

You will have to remount the device(s) with the ext3 option, it will not default to this. You will also have to edit fstab to have them mounted with the ext3 option upon successive reboots.

Cool
 
Old 01-10-2005, 09:10 AM   #11
merlin23
Member
 
Registered: Dec 2004
Location: Vienna
Posts: 46

Original Poster
Rep: Reputation: 15
Hi anybody, I figured something out now:

It's not that my filesystem does not support >2GB file size, because I can make files larger than this actually!
(tried it with "dd if/dev/zero of=/test bs=1M count=3500")

Could it be that the app uses the fseek command, that handles ints, and not longs so there is only 32-bit support???
(I think I've found an issue about htat somewhere)

This would be the only idea I have...

What makes me wonder is that htdig claims to have no size limit in the documentation...
maybe I have to recompile the program against another glibc library?
Does anybody know how to do this??
 
Old 01-11-2005, 09:01 AM   #12
merlin23
Member
 
Registered: Dec 2004
Location: Vienna
Posts: 46

Original Poster
Rep: Reputation: 15
HEELLLLPPP PLEASE !!!!

Please help me !!!!
I don't know what to do...searched the whole net in meanwhile....
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Creating large size file for testing kushalkoolwal Programming 5 03-31-2010 08:56 AM
Very large pdf file size gmgartner Linux - Software 5 04-28-2005 12:23 PM
How to enable large file support (LFS)? maddmike SUSE / openSUSE 0 12-27-2004 11:45 AM
Large File Support on a 32-bit machine JapanMtlExpat Linux - Software 3 11-26-2004 03:50 AM
File does not exist/Large file support dreamtheater Linux - General 3 04-19-2004 10:14 AM


All times are GMT -5. The time now is 04:36 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration