4Tb Server
 

I've been asked to look into the possibility of hosting a 4Tb server for dishing out files through a website. I'm not too keen on a Windows box, due to the licensing limitations, so was thinking about a linux box. Does anyone have any advice, or experience of doing this?

Be careful of the block size an filesystem limits...some distributions (like Red Hat) will only support a block size of 1TB...therefore a filesystem maximum of 1TB. 2TB is possible, but it might be pushing it. Don't try to use all 4Tb on one big filesystem unless you want to expect problems.

I've never built a server on that scale, but most distros will run the apache web server out of the box, so getting ~4Tb's worth of files online isn't much harder than getting them all in one directory structure.

For a that size, I'd have one partition for all of the files [EDIT: given the previous post, check that the distro will support this first. I think most newer kernels will, but it may depend on the filesystem.], and mount it as something like /usr/share/httpdocs, and point your documentroot in httpd.conf to that address. If you're going to be saving files to the web-partition a lot, then use a journeling filesystem (I suggest reiserfs), otherwise don't (use ext2) as you'll use the disk space more efficiently.

If you want to stream over protocols other than HTTP, then Apache won't help you; you'll need to look at a more appropriate server (try searching www.freshmeat.net)

You might also want to consider using software RAID; distros like SuSE will set that up out of the box (in an advanced install), and you can gain speed and/or reliability savings by trading off disk space. RAID (or LVM) will also let you combine multiple partitions into one big one. Hardware RAID is an option, but check the distro's HCL (hardware compatibility list) to make sure that it will actually work (I say that from experience!).

Hardware: Get a box with one processor, and don't go for the fastest you can find. The disk or network speed will probably be the limiting factor. 64-bits won't help you here (unless you want to move the entire 4Tb of data into one tmpfs (in-memory) partition, in which case if you want more than 4Tb of RAM, so you'll get a speed benefit from the increased native address space).

The files to be moved will be 30-300Mb in size. Demand has yet to be established, I think this is more of somebody's pipe dream, rather than something that is needed, but if the money man wants it... It will be about 5 years worth of work going onto the server initially, with more added as it's created. The idea is to produce two identical 4Tb systems, one purely as a backup (seems the only way of backing up something that size cheaply).

Filesystem limits
 

Can someone offer some more clarification on the filesystem size limits?

I am running RH ES3.0 and as stated in this thread, the filesystem size is 1TB. I want to use a Direct Attached Storage unit that is 2TB and expandable to 4.8 TB. It would be using a Fiber Channel or scsi interface. I have previously used a Network Storage device and it didn't have the size limitations but unfortunately it doesn't offer the throughput that I need.

My question is - Does the filesystem size limit of 1TB = Total disk space( server disks + DAS) or can I split the DAS into multiple 1TB volumes in addition to my disk space on my server?

I suggest XFS instead of ReiserFS if you are going to store files with a size of 30 to 300 MB. XFS comes in real handy if using LVM or EVMS because it can handle million terabytes of space when using a 64-bit processor and kernel 2.6.x with big block support. An Opteron system will work well for now and any future upgrades.

I use XFS on my 120 GB hard drives and it has very high data throughput.

I had to go with an XServe system in the end. The files turned out to be some strange customised format that only an Apple server could understand. Netatalk and Windows Appletalk just got totally confused.