That's right, 45TB - here are some of the options I've considered, would like some input:

Debian+JFS - lots of guides on using this vast ammount of storage with debian and JFS, however I view debian as a linux kernel used to fix broken linux systems, also not had much experience with JFS. Command line could be complicated when fixing faulty disks (most people have little experience with faulty disks as it's a command we use so little).

Centos+EXT4 - Again, no native ZFS support, EXT4 is fairly new, not much documentation on such a vast ammount of disk space.

Opensolaris+ZFS - Fairly proficient with solaris, still no gui or documentation, benefits of ZFS make it ideal for such a large filesystem, ZFS also natively supported in the kernel, no weird hacks to get it working, solaris have a similar product which they have working well, however still cli only and read somewhere that multipliers support has only recently been introduced. Iscsi support also natively present.

Openfiler - Never used openfiler, however it has full iscsi support and a gui and I suspect decent documentation, also ZFS present in the kernel, although not a native addition.

FreeNAS + ZFS - little worried as I view freenas as a solution to peoples movie+mp3 storage and not a super efficient system, however it has vast documentation and a strong kernel (freebsd) and ZFS support with iscsi, and a command line, my biggest worry is fixes via the command line, I HATE freebsd and really struggle when using it, initiating those disks could be hard.

Planning on getting a storage pod from backblaze (documentation here. The how to's at the bottom dont specify an operating system but hints at running ZFS on fuse (not ideal).

ZFS has a GUI in the Oracle Web Console + the ZFS Administration applet.

What the heck do you mean by "no native command line running"?

I have no opinion on JFS, FreeNAS, or OpenFiler as I haven't used them. Of the others, ZFS on *Solaris would be the best IMO. Else, ext4 on CentOS 6 (which isn't out yet.. but should be real soon now?). ext4 is old enough now IMO.

The major benefit to ZFS and ext4 in this scenario would be the speedy fsck times (none in the case of ZFS).

1 members found this post helpful.

Typo. I'll run EXT4 in journalled or writeback mode, it won't fsck. Looking at freenas if it has native ZFS support, opensolaris is mysecond choice.

Thanks for your advice.

Scrap that, no RAID 6 on freenas!

No ZFS in openfiler. Looks like it's gonna be opensolaris...

BSD or Solaris for zfs would be my choice. In fact might want to look at openIndiana project. Last looked pretty good. Big learning curve on it though to be useful. BSD's may be a bit easier. Don't forget to think about iSCSI target's.

1 members found this post helpful.

There is a FreeBSD port of ZFS, is there not? I don't personally have experience using the file system, but OpenSolaris isn't looking so good after the Oracle takeover. FreeBSD may be a safer bet.

1 members found this post helpful.

I'm not at all concerned about the future of opensolaris, oracle are continuing support for it (after lots of dispute). My biggest concern is multiplier support; it's only recently been written into the kernel, which means there's a good chance whatever RAID cards I pick won't be supported. Someone recommended nexenta to me (opensolaris with debian style commands) could be well suited given that it's slightly more user friendly than solaris (not really a concern for me but could be for new users). Still doesn't address my concerns with sata multipliers though.

I think ZFS should be my prime focus here, slightly fearful of running BSD with ZFS as my understanding is that ZFS is a propriety (i.e code owned by oracle) and thus anything but solaris will be hacks of some kind. I've read some stories about people putting ZFS fuse filesystem on top of things like red hat and hitting major performance issues.

Trying to avoid anything with a big learning curve as support for this kind of thing will be patchy, out of the guides I've read; mostly people are using debian and JFS, so at least if I run into problems I know people have a similar setup... I just think debian and JFS isn't a brilliant setup really. I'll probably end up trying lots of OSs, it's a real bity about freeNAS not having RAID 6 support, would of been absolutely ideal for my needs.

Try this maybe. http://openindiana.org/

As to the code of ZFS, it is not exactly free but in fact really usable. http://en.wikipedia.org/wiki/Common_...bution_License ZFS on BSD I believe to be real ZFS and not a hack at all.

1 members found this post helpful.

That looks really slick, I suspect I'm going to end up trying lots of different combinations: Openfiler, centos, openindiana (or some other solaris based distro) and the most suitable freeBSD distro are all looking promising. Having RAID 6 is the most important thing, followed by usability, and then ZFS and finally extras such as iscsi.

Big thanks for everyones help, this has been a really interesting learning curve, seem like every OS has pitfalls and benefits, where as initially I thought this would be entirely based on preference.

Why the RAID6 importance ?. Have you seen the zfs demos on youtube ?. Pretty impressive.

Faster, easier to manage, maximum redundancy (like to happen given the vast number of disks).

Solaris 10 - not OpenSolaris is available from Oracle/Sun as a free download:

"www(dot)oracle(dot)com/technetwork/server-storage/solaris/downloads/index(dot)html"

It has a nice GUI interface or cli as you choose. RHEL works with ZFS as well, I believe.

The SunFire x4500 shipped with Solaris and ZFS I'm pretty sure...I know they had Solaris on them.
and the 48 500gig drives were part of a cluster at the installation I was at.

Personally, I'd go with Solaris, or OpenSolaris...Oracle may have bought them out but most of the people are still the same.

Good Luck!

It's the same problem with solaris and open solaris and solaris express and anything built on the solaris kernel - lack of support for sata multipliers. The sunfireX4500 had a custom built motherboard without sata multipliers. From what I understand redhat has no support for ZFS (I haven't thoroughly investigated this yet). Running ZFS on fuse apparently results in really crap/boarder line unusable performance.

I'll give some form of opensolaris a shot, the indianna thing mentioned previously looks like it may have more support for 'other' hardware so maybe worth a shot. I'm fairly competant with solaris products so I'm more than happy using it over (most) other unix/linux systems (exception being redhat, centos and possibly ubuntu).

We've been using XFS on our 40TB RAID 5 arrays with no problem. Any reason XFS is not in your choices?