Solution for storage purposes
 

Hi guys,

I am planning for a storage solution for a media company, for them to store large amount of their magazine work on it. Estimated required space per year is almost 3TB.

I'm thinking is it even feasible to use linux as a base operating system with huge-ass amount of data space behind it...

just thinking wildly now.. not sure abt the actual implementation.. hence I thought i ask you guys, to see if there's any suggestions :)

thanks!

is it backing store or online store you need?

for online store access I would recommend SCSI for speed but 3TB could be cost prohibitive

for backing store, why not go for a rack case with USB2 HDDs chained mapped under LVM ... supported natively in latest kernels/distros.

Or you could try RAID with regular SATA drivers, which are cheaper than SCSI ones. But, do not use RAID0, it increases the probability of data loss. Use RAID1 and up.

And maybe you should somehow implement a daemon to periodically read all data - so corrupted sectors get restored using RAID redundancy before some more data about the same data block is lost.

I have a 2TB backup server here, using IDE drives. I would use RAID5, but I don't know what your requirements are. RAID5 will give you redundancy (if 1 drive fails, your data is not lost as in LVM) and fast reads, but writing is slower since it would be writing to all the drives.
What will your accesses be like? As keratos asked, is it for backup purposes, or will the users be accessing data from it? (honestly, either way I would probably use RAID5 - I hate to lose data, and a mirroring setup takes too many drives IMHO). If a lot of the users (20+) will be mounting the drives via NFS then I would use either a BSD distro or a Solaris distro as those seem to be a little better at handling NFS that Linux (speed wise).
Not sure what the limitations are as far as how many TB's the kernel/file system will handle, but if you go extremely large you may want to look at things like iSCSI and fibre channel. Grab a Linux magazine and look at the ads in there for the NAS solutions. Most of the larger companies I do work for end up using a NAS box of one form or another.

What's your budget and what are the access requirements? Are users needing to access such data? Are you planning this as a large file server? What type of files are we talking about? I'd imagine magazine as in articles and a lot of text right? If that's the case, you'll probably have a lot of duplication involved. You may want to look into deduplication products. This requires less physical drive space that crams a lot of data by de-duping it, etc. Other than that and if you want the generic option, a SAN or NAS is the only option if you know you'll be growing at 3TB a year. Also with this amount of data, I hope you have a backup plan in place.