Scalable, cheap disk storage? (Hypothetical, interested in ideas)
Linux - GeneralThis Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
Scalable, cheap disk storage? (Hypothetical, interested in ideas)
At the moment this is completley hypothetical and i'm just interested in your ideas and feedback, and also expanding my knowledge on the subject:
I'm wondering how you would go about creating a scalable and cheap (ie: cheaper than a SAN or massive SCSI array) array of SATA disks. Currently I have a box at home with 7 x 320GB SATA disks arranged in a RAID-5 array, giving about 1.9TB formatted. However, to resize an array of this size is headache inducing to say the least (thank god for UPS's!)
I'm interested in a solution that involves several machines and is scaleable, so I can add more nodes at a later date. It should be some sort of storage cluster, as it were. It doesn't necessarily have to actually be one big disk array internally, but from the outside (Samba access), it should appear as one "disk/folder". My thoughts on the matter is having several "slave nodes" which are much like my current system and simply have a large capacity RAID array on them, and then having a "master node" that can merge all the array's into one logical "drive", or use the storage provided by these arrays, and use some sort of filesystem (such as MogleFS to store the files, while using Samba to serve them up across the network.
What are your ideas? If I need to clarify my "requirements", let me know
I would use a hardware RAID contrroller that allows hot swapping and array expansion on the fly such as the 3ware controllers.. add drive,issue expand array command, walk away... job is done.
otherwise I suppose a NAS solution using something like NASLite or freeNAS would work with multiple machines..
Personally, the idea of having something that appears to be one "folder" at 1.9 Tb scares the sh** out of me. It seems like a point for too much failure, and too much misuse if it's anything other than a one-user system.
IMHO, anything that combines that much space into one logical "place" is just adding unneeded code, lag time, and complexity. If they're 320Gb drives, why not just have hda1, hda2, hda3, etc. in the old-fashioned way, mount them to folders with meaningful names, and call it a day?
On the useful side of advice, if you're storing that much data, you need a strong, reliable, fault-tolerant file system. I would recommend against ext2, or any other Linux default. though they may be journalling, they're not the best out there in terms of crash-worthiness.
I'd suggest OpenSolaris or one of the BSD systems (FreeBSD or NetBSD).
There's also a distro called FreeNAS which is a NAS package based on FreeBSD, and has a strong filesystem.
Sounds like LVM2 at http://evms.sf.net is what you're looking for, it features scalable logical volumes, resizable partitions, snap copies, bad block relocation, partition table types that support > 2TB partitions, most of the modern file systems and a lot more. It works well with HA-clustering and as an iSCSI target.
IMHO, anything that combines that much space into one logical "place" is just adding unneeded code, lag time, and complexity. If they're 320Gb drives, why not just have hda1, hda2, hda3, etc. in the old-fashioned way, mount them to folders with meaningful names, and call it a day?
Because those won't give him the a) kick and b) redundancy
that a RAID-5 system does. Striping the data over several disks
IMPROVES performance, it doesn't deteriorate it (as a rule of
thumb, anyway. Happy to discuss the differences it makes to
e.g. an Oracle installation to have more disks laid out differently,
but I wouldn't want to miss out on redundancy in either way.)
Another good reason for wanting such large space is e.g. for
movie-remastering in a larger scale :}
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.