LinuxQuestions.org
View the Most Wanted LQ Wiki articles.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - General
User Name
Password
Linux - General This Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.

Notices

Reply
 
Search this Thread
Old 07-21-2004, 06:01 PM   #1
nkendrick
LQ Newbie
 
Registered: Nov 2003
Posts: 16

Rep: Reputation: 0
Very confused about 2GB file size limit!


Apologies if this is old ground but having spent the last hour or so searching this site and the Internet in general I am none the wiser and I would appreciate someone's experience on this:

I am running White Box Enterprise Linux, which is a generic version of Red Hat Enterprise Linux 3 (2.4 kernel). I am using the ext3 file system on all partitions.

I need to take a regular copy of a file located on a Windows 2000 server; it's a dump of a very large MSSQL database and it's currently about 2.6GB in size and likely to grow. I want to use the dar backup utility to take the copy.

If I 'look' at the file via a samba mount, the file claims to be 8E (Exobytes) big so I guess I'm hitting a problem with large file support. (??)

So my questions are:

Does ext3 natively support this file size?

Is this a just a Samba 2GB limitation or a combination of both Samba and ext3?

I assume I need to enable large file support for either ext3 and/or Samba.

I have also read about the Samba upgrade to CIFS.

So, what do I have to do to allow me to store large files on the Limux server (if anything) and also be able to see and copy them successfully via Samba (CIFS?)

Have you any pointers to sites that explain everything in very simple terms - this would be really appreciated because so far I am trying to patch together an answer from various sites that refer to different distros and I am getting rather confused

Many thanks
 
Old 07-21-2004, 07:42 PM   #2
jschiwal
Guru
 
Registered: Aug 2001
Location: Fargo, ND
Distribution: SuSE AMD64
Posts: 15,733

Rep: Reputation: 655Reputation: 655Reputation: 655Reputation: 655Reputation: 655Reputation: 655
Before I forget, thank you for introducting me to 'dar'.

I believe that originally that the ext3 file system had a 2 GB file size limit. I've also read the older glibc routines used signed integers giving this limit also. The same message said that some programs would use the newer glibc routines improperly. There was a 2GB limit with 2.2 kernels.
If programs like 'ls' are compiled using signed 32-bit integers, there is a 2048MB limit in file size. Unsigned 32-bit integers are limited to 4096MB

Linux Gazette article on Journal File Systems

I think you need to look at your versions of:
The kernel
The glibc library
Samba and related libraries.

You may want to use the 'ldd' command to see which library versions your clients are built with. (assuming the information isn't stripped for static libraries).

Look in the Samba web site with the particular version of your client and server. They may have posted limits.

---

Another thing you could do is use the 'dd' command and try to create a 5 GB file on your file system (assuming you have the space). This could eliminate whether the ext3 version you are using has a 2 GB limit.

----

I believe that the problem may be in the version of samba you are using. But this is only my gut feeling. The original SMB protocol used 32 bit integers for file size.

Look at section 3.1.3
http://us1.samba.org/samba/ftp/specs/smb-nt01.doc

excerpted form smbpub.txt --> http://luna.gui.uva.es/sites/samba.org/specs/smbpub.txt
Code:
ULONG FileSize;
File size

USHORT Reserved [5];
Reserved - client should ignore

USHORT ByteCount;
Count of data bytes = 0


FileAttributes are as described in the Attributes Encoding section of this document.

Note that FileSize is limited to 32 bits, this request is inappropriate for files whose size is too large.
SET_INFORMATION: Set File Attributes
This message is sent to change the information about a file.
Client Request
Description
j

So the version of the smb protocol used between the two machines may be another factor.


----

Good luck!

Last edited by jschiwal; 07-21-2004 at 09:24 PM.
 
Old 07-22-2004, 04:49 AM   #3
nkendrick
LQ Newbie
 
Registered: Nov 2003
Posts: 16

Original Poster
Rep: Reputation: 0
Thanks for the reply - I will check out what you suggest.

dar is a really useful tool!
 
Old 08-15-2005, 11:07 PM   #4
liyuefu
LQ Newbie
 
Registered: Jun 2005
Posts: 16

Rep: Reputation: 0
I tested it on RHEL4, the problem is solved.

I made a test on RHEL4U1.

I mount a fat32 filesystem to linux, and I can create file whose size is great than 2G,
I tested it using 3.2G, and it worked.

I am not sure from which version the kernel begins to support large file, but RHEL4 is just fine.
its kernel is 2.6.9-11
 
Old 08-25-2005, 02:37 AM   #5
lucky_dev36
Member
 
Registered: Jan 2005
Distribution: Debian
Posts: 34

Rep: Reputation: 15
2.1 GB

I wrote a python script to test this. I stooped it on purpose when it got to 2.1G thinking it would lock the file. Nope... It worked. My Slackware stopped it at just under 2G. It is distro depenent (or glibc dependent). Maybe it's gnu libc's typedef's. (probably using int.)

[Theory]

Large files are supported by turning int into unsigned int, ehich can hold more data.

[/Theory]
 
Old 08-25-2005, 12:26 PM   #6
lucky_dev36
Member
 
Registered: Jan 2005
Distribution: Debian
Posts: 34

Rep: Reputation: 15
21.4 G's!!!

I made a 21.4 GB File overnight y executing my python script!
 
Old 09-29-2005, 08:14 PM   #7
artois_val
Member
 
Registered: Jan 2004
Posts: 35

Rep: Reputation: 15
Can we have your python script ?

Please can you share it ?

Thanks in advance
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Redhat 9.0 >2GB file limit Wilbs Red Hat 2 08-12-2004 02:26 PM
2GB File Limit AS2.1 jbovaird Red Hat 19 11-14-2003 12:29 PM
mencoder file size limit 2GB chii-chan Linux - Software 3 10-22-2003 05:27 AM
2.4.20 Kernel has file size limit=2GB?? How can this be! digizer0 Linux - General 7 01-14-2003 11:00 PM
Is there a way round the 2gb File size Limit smurf Linux - General 2 07-10-2002 03:57 PM


All times are GMT -5. The time now is 02:35 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration