LinuxQuestions.org
Latest LQ Deal: Complete CCNA, CCNP & Red Hat Certification Training Bundle
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Distributions > Slackware
User Name
Password
Slackware This Forum is for the discussion of Slackware Linux.

Notices


Reply
  Search this Thread
Old 03-17-2017, 03:58 AM   #1
kikinovak
MLED Founder
 
Registered: Jun 2011
Location: Montpezat (South France)
Distribution: Slackware, MLED, MLES, CentOS
Posts: 3,295

Rep: Reputation: 1824Reputation: 1824Reputation: 1824Reputation: 1824Reputation: 1824Reputation: 1824Reputation: 1824Reputation: 1824Reputation: 1824Reputation: 1824Reputation: 1824
[Slightly OT] Robust file storage and archiving system


Hi,

I've been contacted recently by a rather big company, a major player in the broadcasting business, who's looking for "a robust and secure free storage and archiving solution". Well, the Linux and BSD world has quite some solutions to offer here, but I have an extra requirement that is going to complicate things... or simplify them, as is. Let me explain.

The company wants me to offer some training for said solution for their admin staff, with courses that can take up to five days. After five days, the admins should be ready to deploy the systems in the company.

Well, I've done some extensive Linux training in the past few years, with courses mostly based on Slackware - due to its pedagogic virtues - and then presenting other distributions like CentOS and Debian with their differences. The problem is, these courses are usually designed to go over several months, something like forty full days of training over four months. Seen from this angle, the equation seems impossible to resolve.

But then I remembered that a few years ago I experimented with the FreeNAS distribution. As some of you may know, FreeNAS is a storage and archiving solution based on FreeBSD and - currently - the ZFS file system. I downloaded it, played around a bit on a spare sandbox server, and it looks like this could do it. The FreeNAS documentation is as excellent as the FreeBSD handbook. And it looks much more feasible to organize a five-day training course on the specific topic of FreeNAS than to teach Linux basics, Bash scripting and server and security basics from scratch. Back in 2010 I made such a five-day crash course on Oracle Linux for the french motorway company, and after five days of pure mental sauna, I swore to myself that I would never do this again (though the students seemed to enjoy the course).

Anyway, I would be curious to have your views on FreeNAS if some of you happen to use it. The reason I never adopted it are mostly because I always use Slackware on my own servers and those of my clients, and the small Synology NAS I have in my office has only a small FTP server on it, and I only use it to store Ghost4Linux system images.

Cheers from the sunny South of France,

Niki
 
Old 03-17-2017, 04:10 AM   #2
Turbocapitalist
Senior Member
 
Registered: Apr 2005
Distribution: Ubuntu, Devuan, OpenBSD
Posts: 1,618
Blog Entries: 3

Rep: Reputation: 692Reputation: 692Reputation: 692Reputation: 692Reputation: 692Reputation: 692
It depends on what they mean by 'archiving' so you may want to pin them down on that before investing heavily in any particular path of exploration. FreeNAS does not do any archiving in the formal use of the term, just storage. ZFS does provide a lot of robustness in regards to bitrot or gradual drive deterioration, especially if used with redundant drives.

DSpace, which could run fine on top of ZFS, provides a traditional digital repository aka archive. Hydra is another one. I'm not up on the latest developments, but those two project would provide a good place to start.
 
Old 03-17-2017, 05:36 PM   #3
Gerard Lally
Senior Member
 
Registered: Sep 2009
Location: Ireland
Distribution: Slackware64, Crux64, NetBSD64
Posts: 1,118

Rep: Reputation: 658Reputation: 658Reputation: 658Reputation: 658Reputation: 658Reputation: 658
If this broadcasting company intends to store and archive huge media files on this system you should consider the xfs filesystem. Couple xfs for storage with rsnapshot for archival, and backup offsite to a zfs or hammer system to pre-empt bitrot. This won't take 5 days to teach, even if you do use shell scripts.

Last edited by Gerard Lally; 03-17-2017 at 05:37 PM.
 
Old 03-18-2017, 02:38 AM   #4
kikinovak
MLED Founder
 
Registered: Jun 2011
Location: Montpezat (South France)
Distribution: Slackware, MLED, MLES, CentOS
Posts: 3,295

Original Poster
Rep: Reputation: 1824Reputation: 1824Reputation: 1824Reputation: 1824Reputation: 1824Reputation: 1824Reputation: 1824Reputation: 1824Reputation: 1824Reputation: 1824Reputation: 1824
Quote:
Originally Posted by Gerard Lally View Post
If this broadcasting company intends to store and archive huge media files on this system you should consider the xfs filesystem. Couple xfs for storage with rsnapshot for archival, and backup offsite to a zfs or hammer system to pre-empt bitrot. This won't take 5 days to teach, even if you do use shell scripts.
Don't forget that in that case, I'll have to teach complete newbies who mostly have never touched a Linux system. I do teach Linux courses for newbies, from scratch, but these courses take (much) more than five days. If you're focusing a course on the command line from scratch, five days allow you to barely cover the basics (e. g. roughly what you'll find in the Slackbook).

On a side note, I've spent the last three years writing a book about Linux Administration from scratch for the biggest french tech editor (Eyrolles), based on Slackware. We've finished production, and after several delays, it's finally due for April. It's the first one of two volumes, counting more than five hundred pages, and this is only covering the basics. We're not even tackling the server stuff. In my experience, this is downright impossible to work through in five days.
 
Old 03-18-2017, 08:14 AM   #5
Gerard Lally
Senior Member
 
Registered: Sep 2009
Location: Ireland
Distribution: Slackware64, Crux64, NetBSD64
Posts: 1,118

Rep: Reputation: 658Reputation: 658Reputation: 658Reputation: 658Reputation: 658Reputation: 658
Quote:
Originally Posted by kikinovak View Post
Don't forget that in that case, I'll have to teach complete newbies who mostly have never touched a Linux system. I do teach Linux courses for newbies, from scratch, but these courses take (much) more than five days. If you're focusing a course on the command line from scratch, five days allow you to barely cover the basics (e. g. roughly what you'll find in the Slackbook).

On a side note, I've spent the last three years writing a book about Linux Administration from scratch for the biggest french tech editor (Eyrolles), based on Slackware. We've finished production, and after several delays, it's finally due for April. It's the first one of two volumes, counting more than five hundred pages, and this is only covering the basics. We're not even tackling the server stuff. In my experience, this is downright impossible to work through in five days.
Well they shouldn't demand a robust solution, in that case. A robust solution for archival is something I'd expect to be in place decades later. If they just want something they can teach their Windows admins they should look elsewhere.
 
Old 03-18-2017, 08:33 AM   #6
kikinovak
MLED Founder
 
Registered: Jun 2011
Location: Montpezat (South France)
Distribution: Slackware, MLED, MLES, CentOS
Posts: 3,295

Original Poster
Rep: Reputation: 1824Reputation: 1824Reputation: 1824Reputation: 1824Reputation: 1824Reputation: 1824Reputation: 1824Reputation: 1824Reputation: 1824Reputation: 1824Reputation: 1824
Quote:
Originally Posted by Turbocapitalist View Post
ZFS does provide a lot of robustness in regards to bitrot or gradual drive deterioration, especially if used with redundant drives.
I did some more reading about ZFS, and I must say I like it a lot. I installed FreeNAS in a VM for testing purposes, and it looks very clean. A clean web interface on top of FreeBSD, which I already know a bit. Their documentation is excellent, and it seems like the system itself is well maintained, with more than 50 developers and a very active community. My gut feeling on this is very positive, and I guess I'll spend a couple weeks experimenting with this.

Cheers,

Niki
 
Old 03-18-2017, 08:45 AM   #7
Turbocapitalist
Senior Member
 
Registered: Apr 2005
Distribution: Ubuntu, Devuan, OpenBSD
Posts: 1,618
Blog Entries: 3

Rep: Reputation: 692Reputation: 692Reputation: 692Reputation: 692Reputation: 692Reputation: 692
ZFS is also apparently available for Slackware, that gives you an edge, though in Linux the support is a bit behind FreeBSD.

If you're just looking for storage then a dead simple way is to just use SFTP instead of fighting with a web interface. It lowers the complexity of setup and maintenance, and thus costs. Regular GNU/Linux users can use their file managers to transfer. OS X have it a little harder, but can use FileZilla or, with the help of sshfs, use the Finder. Legacy systems have WinSCP and FileZilla. I'm not sure about using WinSCP or FileZilla with keys, but sshfs and the GNU/Linux file managers all work nicely with keys. It's easy to set up, simple to maintain, and takes little training to use once set up (even with keys). An added bonus of SFTP with keys is that it is more or less safe to put out on the public net.

I suppose, if there is enough drive space you could use rsync on the storage device to take nightly snapshots locally. The --link-dest option can be used for incremental backups, and of course backup are essential, but you might also keep versions available from yesterday or last week available on the same machine.

Last edited by Turbocapitalist; 03-18-2017 at 08:47 AM.
 
Old 03-18-2017, 11:48 AM   #8
kikinovak
MLED Founder
 
Registered: Jun 2011
Location: Montpezat (South France)
Distribution: Slackware, MLED, MLES, CentOS
Posts: 3,295

Original Poster
Rep: Reputation: 1824Reputation: 1824Reputation: 1824Reputation: 1824Reputation: 1824Reputation: 1824Reputation: 1824Reputation: 1824Reputation: 1824Reputation: 1824Reputation: 1824
Quote:
Originally Posted by Turbocapitalist View Post
ZFS is also apparently available for Slackware, that gives you an edge, though in Linux the support is a bit behind FreeBSD.

If you're just looking for storage then a dead simple way is to just use SFTP instead of fighting with a web interface. It lowers the complexity of setup and maintenance, and thus costs. Regular GNU/Linux users can use their file managers to transfer. OS X have it a little harder, but can use FileZilla or, with the help of sshfs, use the Finder. Legacy systems have WinSCP and FileZilla. I'm not sure about using WinSCP or FileZilla with keys, but sshfs and the GNU/Linux file managers all work nicely with keys. It's easy to set up, simple to maintain, and takes little training to use once set up (even with keys). An added bonus of SFTP with keys is that it is more or less safe to put out on the public net.

I suppose, if there is enough drive space you could use rsync on the storage device to take nightly snapshots locally. The --link-dest option can be used for incremental backups, and of course backup are essential, but you might also keep versions available from yesterday or last week available on the same machine.
In cases where I'm the admin, I have various solutions that I've been using in production for quite some time. In mixed environments (Windows and OS X clients) I usually setup Samba on a Slackware server with software RAID 1 or 5 (depending on the number of hard disks), and then on the backup machine I recently replaced my handmade rsync-over-ssh script by rsnapshot, which runs great. (Sometimes you reinvent the wheel before discovering someone made a wheel before you.)

But in this special case, FreeNAS would allow me to train a staff of newbie admins to setup some machines. When a client asks me for a passport identity photo, I won't sell them a bigger-than-life oil painting.
 
Old 03-18-2017, 01:02 PM   #9
Turbocapitalist
Senior Member
 
Registered: Apr 2005
Distribution: Ubuntu, Devuan, OpenBSD
Posts: 1,618
Blog Entries: 3

Rep: Reputation: 692Reputation: 692Reputation: 692Reputation: 692Reputation: 692Reputation: 692
Quote:
Originally Posted by kikinovak View Post
But in this special case, FreeNAS would allow me to train a staff of newbie admins to setup some machines. When a client asks me for a passport identity photo, I won't sell them a bigger-than-life oil painting.
Yes, you know the scope of the project. The turn-key nature of FreeNAS is appealing for newbie admins. I'm a fan of keeping it simple and Samba always strikes me as complicated so I avoid it, though I used to use it, especially since it can only stay on a LAN and not a hostile LAN at that. With enough effort Samba could be made available over OpenVPN. However, FreeNAS does support SFTP so I would say to give it a thought for the reasons that a) SFTP is secure and easy to use, and b) later they can access remotely if they decide to.

What other constraints are there?
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Is SD solid enough as primary storage for file system in embedded system ? 5883 Linux - Embedded & Single-board computer 5 09-30-2012 01:27 PM
LXer: SDFS: A Robust Deduplication File System for Linux LXer Syndicated Linux News 0 03-26-2010 10:50 AM
Archiving old files from /home/* directories to less expensive disk storage. Cerephim Linux - Server 6 11-06-2009 11:25 AM
[SOLVED] cp command, problems with archiving from only one file system... s2cuts Linux - General 5 11-02-2009 07:58 AM
LXer: PeaZip: Robust But Easy OSS File Management, Compression and Archiving LXer Syndicated Linux News 0 11-13-2008 11:40 PM


All times are GMT -5. The time now is 12:25 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration