LinuxQuestions.org - Very confused about 2GB file size limit!

- Linux - General (https://www.linuxquestions.org/questions/linux-general-1/)

- - Very confused about 2GB file size limit! (https://www.linuxquestions.org/questions/linux-general-1/very-confused-about-2gb-file-size-limit-208014/)

Very confused about 2GB file size limit!

Apologies if this is old ground but having spent the last hour or so searching this site and the Internet in general I am none the wiser and I would appreciate someone's experience on this:

I am running White Box Enterprise Linux, which is a generic version of Red Hat Enterprise Linux 3 (2.4 kernel). I am using the ext3 file system on all partitions.

I need to take a regular copy of a file located on a Windows 2000 server; it's a dump of a very large MSSQL database and it's currently about 2.6GB in size and likely to grow. I want to use the dar backup utility to take the copy.

If I 'look' at the file via a samba mount, the file claims to be 8E (Exobytes) big so I guess I'm hitting a problem with large file support. (??)

So my questions are:

Does ext3 natively support this file size?

Is this a just a Samba 2GB limitation or a combination of both Samba and ext3?

I assume I need to enable large file support for either ext3 and/or Samba.

I have also read about the Samba upgrade to CIFS.

So, what do I have to do to allow me to store large files on the Limux server (if anything) and also be able to see and copy them successfully via Samba (CIFS?)

Have you any pointers to sites that explain everything in very simple terms - this would be really appreciated because so far I am trying to patch together an answer from various sites that refer to different distros and I am getting rather confused :confused:

Many thanks

Before I forget, thank you for introducting me to 'dar'.

I believe that originally that the ext3 file system had a 2 GB file size limit. I've also read the older glibc routines used signed integers giving this limit also. The same message said that some programs would use the newer glibc routines improperly. There was a 2GB limit with 2.2 kernels.
If programs like 'ls' are compiled using signed 32-bit integers, there is a 2048MB limit in file size. Unsigned 32-bit integers are limited to 4096MB

Linux Gazette article on Journal File Systems

I think you need to look at your versions of:
The kernel
The glibc library
Samba and related libraries.

You may want to use the 'ldd' command to see which library versions your clients are built with. (assuming the information isn't stripped for static libraries).

Look in the Samba web site with the particular version of your client and server. They may have posted limits.

---

Another thing you could do is use the 'dd' command and try to create a 5 GB file on your file system (assuming you have the space). This could eliminate whether the ext3 version you are using has a 2 GB limit.

----

I believe that the problem may be in the version of samba you are using. But this is only my gut feeling. The original SMB protocol used 32 bit integers for file size.

Look at section 3.1.3
http://us1.samba.org/samba/ftp/specs/smb-nt01.doc

excerpted form smbpub.txt --> http://luna.gui.uva.es/sites/samba.org/specs/smbpub.txt

Code:

ULONG FileSize;

File size



USHORT Reserved [5];

Reserved - client should ignore



USHORT ByteCount;

Count of data bytes = 0





FileAttributes are as described in the Attributes Encoding section of this document.



Note that FileSize is limited to 32 bits, this request is inappropriate for files whose size is too large.

SET_INFORMATION: Set File Attributes

This message is sent to change the information about a file.

Client Request

Description

j

So the version of the smb protocol used between the two machines may be another factor.

----

Good luck!

Thanks for the reply - I will check out what you suggest.

dar is a really useful tool!

I tested it on RHEL4, the problem is solved.

I made a test on RHEL4U1.

I mount a fat32 filesystem to linux, and I can create file whose size is great than 2G,
I tested it using 3.2G, and it worked.

I am not sure from which version the kernel begins to support large file, but RHEL4 is just fine.
its kernel is 2.6.9-11

I wrote a python script to test this. I stooped it on purpose when it got to 2.1G thinking it would lock the file. Nope... It worked. My Slackware stopped it at just under 2G. It is distro depenent (or glibc dependent). Maybe it's gnu libc's typedef's. (probably using int.)

[Theory]

Large files are supported by turning int into unsigned int, ehich can hold more data.

[/Theory]

I made a 21.4 GB File overnight y executing my python script!

Can we have your python script ?

Please can you share it ?

Thanks in advance