We’ve run into an issue in which a customer is trying to set up an 8TB ext3 filesystem. All of the literature I’ve read thus far (including http://www.redhat.com/rhel/compare/) seems to indicate that this is possible, but the customer’s experience appears to contradict this:

“The 4TB limit with Redhat EL4 is not due to limitations with LVM. The problem is the ext3 filesystem in EL4 has a limit of 4TB per file system. We have attempted to create larger filesystems using ext3, and we consistently hit the 4TB limit.”

Can anyone think of any kernel options or other factors that might be keeping this user from being able to exploit the full filesystem capability? Thanks!

I don't know the answer to that, but sounds like that user is a good candidate for ZFS.

Isn't ZFS a Sun product? This is a Red Hat environment.

Yeah, I believe at the moment ZFS is exclusively a Sun product, so a change in OS regimes would be required.. and perhaps it's reasonable for the customer if they're simply setting up a file server. If it's more than a file server, then I don't know what to suggest. Sorry that's my best answer.. hopefully you get a better one.

Have you looked at xfs or jfs?

I wouldn't say it's a bad answer - heck, at this point the customer might prefer a Sun solution over a Red Hat one - but until we hear otherwise we're looking to leverage existing RAID devices running on HP hardware running RHEL 4. In any event, thanks for the response.

poblano

You could just side step the LVM/ext3 issue by putting the mount point of a second LVM within the first LVM. As I personally hold a grudge with LVM (it is great until it blows) this is the way I get near LVM functionality without a LVM.

I suspect I'm not asking the question correctly. I'm wondering why a kernel (2.6.9) and filesystem (ext3) that should support sizes up to 8TB is bumping into a hard limit of 4TB. Are there other variables I should be considering?

Poblano

There are so few people out there using even 1TB you are going to have a limited pool of people who will even be able to guess at what is wrong. This is one of those things your really need to use your RH support for. You paid for it, so why not use it?

To be honest, it's the first time I've dealt with data sets this large. I still remember 5 1/2 inch floppies! Anyway, I ran this question past our Red Hat TAM yesterday, but I suspect he's doing damage control over the recent Linux kernel vulnerability and hasn't had time to respond.

Ah, you young pup you. You mean you weren't around for the good old bendy 8 inch floppies?

It may (or may not) be something that is fixed in later kernels. You have to be careful with RH's kernel numbers. They have long been known to back port patches into older kernel numbers. What this means that even though a RH kernel's number may appear older than a current kernel, it may be patched (by RH) right up to current. I really wish they would quit doing this becuase it makes it a real PITA to figure out what is and what is not patched in their kernel.


Just for curiosity, what kind of files do you need to span across more than 4TB?

I didn't use 8-inch floppies, but I did write IBM 370 Assembly code on keypunch cards back in the day.

This backporting has caused the recent vmsplice vulnerability, introduced with the 2.6.17 kernel, to be seen as far back as the 2.6.9 kernel. It's a gift that keeps on giving!

In this case it's a backup set. My company makes a data warehousing appliance, and we have a customer whose database is over 20TB and expects a compressed backup to be between 4 and 8TB.

Until you get a response from RH can you just split the backup into sections? I would think that one huge backup would be bad in general.

That hardware configuration might be feasible for Nexenta, which supports ZFS. It's still an OS change, but mainly just the kernel. It's designed to enable most of the GNU stuff to run on OpenSolaris, on an x86 or the like.

To support the RH kernel version thingie see for instance this 2.6.17 ext3_fsblk_t patch by Mingming Cao, one of the Extn developers. From the comments you see Ext3 can do 8TB (and over) @4KB blocksize since. * BTW if they *are* able to create an 8TB Ext3 on another system it would be interesting to compare kernel versions, mkfs and tune2fs info.