[SOLVED] [Slightly OT] Robust file storage and archiving system

Hi,

I've been contacted recently by a rather big company, a major player in the broadcasting business, who's looking for "a robust and secure free storage and archiving solution". Well, the Linux and BSD world has quite some solutions to offer here, but I have an extra requirement that is going to complicate things... or simplify them, as is. Let me explain.

The company wants me to offer some training for said solution for their admin staff, with courses that can take up to five days. After five days, the admins should be ready to deploy the systems in the company.

Well, I've done some extensive Linux training in the past few years, with courses mostly based on Slackware - due to its pedagogic virtues - and then presenting other distributions like CentOS and Debian with their differences. The problem is, these courses are usually designed to go over several months, something like forty full days of training over four months. Seen from this angle, the equation seems impossible to resolve.

But then I remembered that a few years ago I experimented with the FreeNAS distribution. As some of you may know, FreeNAS is a storage and archiving solution based on FreeBSD and - currently - the ZFS file system. I downloaded it, played around a bit on a spare sandbox server, and it looks like this could do it. The FreeNAS documentation is as excellent as the FreeBSD handbook. And it looks much more feasible to organize a five-day training course on the specific topic of FreeNAS than to teach Linux basics, Bash scripting and server and security basics from scratch. Back in 2010 I made such a five-day crash course on Oracle Linux for the french motorway company, and after five days of pure mental sauna, I swore to myself that I would never do this again (though the students seemed to enjoy the course).

Anyway, I would be curious to have your views on FreeNAS if some of you happen to use it. The reason I never adopted it are mostly because I always use Slackware on my own servers and those of my clients, and the small Synology NAS I have in my office has only a small FTP server on it, and I only use it to store Ghost4Linux system images.

Cheers from the sunny South of France,

Niki

It depends on what they mean by 'archiving' so you may want to pin them down on that before investing heavily in any particular path of exploration. FreeNAS does not do any archiving in the formal use of the term, just storage. ZFS does provide a lot of robustness in regards to bitrot or gradual drive deterioration, especially if used with redundant drives.

DSpace, which could run fine on top of ZFS, provides a traditional digital repository aka archive. Hydra is another one. I'm not up on the latest developments, but those two project would provide a good place to start.

If this broadcasting company intends to store and archive huge media files on this system you should consider the xfs filesystem. Couple xfs for storage with rsnapshot for archival, and backup offsite to a zfs or hammer system to pre-empt bitrot. This won't take 5 days to teach, even if you do use shell scripts.

Originally Posted by Gerard Lally If this broadcasting company intends to store and archive huge media files on this system you should consider the xfs filesystem. Couple xfs for storage with rsnapshot for archival, and backup offsite to a zfs or hammer system to pre-empt bitrot. This won't take 5 days to teach, even if you do use shell scripts.

Don't forget that in that case, I'll have to teach complete newbies who mostly have never touched a Linux system. I do teach Linux courses for newbies, from scratch, but these courses take (much) more than five days. If you're focusing a course on the command line from scratch, five days allow you to barely cover the basics (e. g. roughly what you'll find in the Slackbook).

On a side note, I've spent the last three years writing a book about Linux Administration from scratch for the biggest french tech editor (Eyrolles), based on Slackware. We've finished production, and after several delays, it's finally due for April. It's the first one of two volumes, counting more than five hundred pages, and this is only covering the basics. We're not even tackling the server stuff. In my experience, this is downright impossible to work through in five days.

Originally Posted by kikinovak Don't forget that in that case, I'll have to teach complete newbies who mostly have never touched a Linux system. I do teach Linux courses for newbies, from scratch, but these courses take (much) more than five days. If you're focusing a course on the command line from scratch, five days allow you to barely cover the basics (e. g. roughly what you'll find in the Slackbook). On a side note, I've spent the last three years writing a book about Linux Administration from scratch for the biggest french tech editor (Eyrolles), based on Slackware. We've finished production, and after several delays, it's finally due for April. It's the first one of two volumes, counting more than five hundred pages, and this is only covering the basics. We're not even tackling the server stuff. In my experience, this is downright impossible to work through in five days.

Well they shouldn't demand a robust solution, in that case. A robust solution for archival is something I'd expect to be in place decades later. If they just want something they can teach their Windows admins they should look elsewhere.

Originally Posted by Turbocapitalist ZFS does provide a lot of robustness in regards to bitrot or gradual drive deterioration, especially if used with redundant drives.

I did some more reading about ZFS, and I must say I like it a lot. I installed FreeNAS in a VM for testing purposes, and it looks very clean. A clean web interface on top of FreeBSD, which I already know a bit. Their documentation is excellent, and it seems like the system itself is well maintained, with more than 50 developers and a very active community. My gut feeling on this is very positive, and I guess I'll spend a couple weeks experimenting with this.

Cheers,

Niki

ZFS is also apparently available for Slackware, that gives you an edge, though in Linux the support is a bit behind FreeBSD.

If you're just looking for storage then a dead simple way is to just use SFTP instead of fighting with a web interface. It lowers the complexity of setup and maintenance, and thus costs. Regular GNU/Linux users can use their file managers to transfer. OS X have it a little harder, but can use FileZilla or, with the help of sshfs, use the Finder. Legacy systems have WinSCP and FileZilla. I'm not sure about using WinSCP or FileZilla with keys, but sshfs and the GNU/Linux file managers all work nicely with keys. It's easy to set up, simple to maintain, and takes little training to use once set up (even with keys). An added bonus of SFTP with keys is that it is more or less safe to put out on the public net.

I suppose, if there is enough drive space you could use rsync on the storage device to take nightly snapshots locally. The --link-dest option can be used for incremental backups, and of course backup are essential, but you might also keep versions available from yesterday or last week available on the same machine.

Originally Posted by Turbocapitalist ZFS is also apparently available for Slackware, that gives you an edge, though in Linux the support is a bit behind FreeBSD. If you're just looking for storage then a dead simple way is to just use SFTP instead of fighting with a web interface. It lowers the complexity of setup and maintenance, and thus costs. Regular GNU/Linux users can use their file managers to transfer. OS X have it a little harder, but can use FileZilla or, with the help of sshfs, use the Finder. Legacy systems have WinSCP and FileZilla. I'm not sure about using WinSCP or FileZilla with keys, but sshfs and the GNU/Linux file managers all work nicely with keys. It's easy to set up, simple to maintain, and takes little training to use once set up (even with keys). An added bonus of SFTP with keys is that it is more or less safe to put out on the public net. I suppose, if there is enough drive space you could use rsync on the storage device to take nightly snapshots locally. The --link-dest option can be used for incremental backups, and of course backup are essential, but you might also keep versions available from yesterday or last week available on the same machine.

In cases where I'm the admin, I have various solutions that I've been using in production for quite some time. In mixed environments (Windows and OS X clients) I usually setup Samba on a Slackware server with software RAID 1 or 5 (depending on the number of hard disks), and then on the backup machine I recently replaced my handmade rsync-over-ssh script by rsnapshot, which runs great. (Sometimes you reinvent the wheel before discovering someone made a wheel before you.)

But in this special case, FreeNAS would allow me to train a staff of newbie admins to setup some machines. When a client asks me for a passport identity photo, I won't sell them a bigger-than-life oil painting.

Originally Posted by kikinovak But in this special case, FreeNAS would allow me to train a staff of newbie admins to setup some machines. When a client asks me for a passport identity photo, I won't sell them a bigger-than-life oil painting.

Yes, you know the scope of the project. The turn-key nature of FreeNAS is appealing for newbie admins. I'm a fan of keeping it simple and Samba always strikes me as complicated so I avoid it, though I used to use it, especially since it can only stay on a LAN and not a hostile LAN at that. With enough effort Samba could be made available over OpenVPN. However, FreeNAS does support SFTP so I would say to give it a thought for the reasons that a) SFTP is secure and easy to use, and b) later they can access remotely if they decide to.

What other constraints are there?