Build own RAID server rather than buying NAS solutions?
Linux - ServerThis forum is for the discussion of Linux Software used in a server related context.
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Introduction to Linux - A Hands on Guide
This guide was created as an overview of the Linux Operating System, geared toward new users as an exploration tour and getting started guide, with exercises at the end of each chapter.
For more advanced trainees it can be a desktop reference, and a collection of the base knowledge needed to proceed with system and network administration. This book contains many real life examples derived from the author's experience as a Linux system and network administrator, trainer and consultant. They hope these examples will help you to get a better understanding of the Linux system and that you feel encouraged to try out things on your own.
Click Here to receive this Complete Guide absolutely free.
Build own RAID server rather than buying NAS solutions?
Hi guys I'm faced with a dilemma of whether to build my own RAID and need some advise and experience comments.
Current situation is I already have a QNAP TS-459 Pro+ NAS with RAID 6 configured. And I will have the need to add a new server to function as DR. If I decide to co-locate my new box to be in the same data center as my existing box, the hosting provider will charge $1,687 USD for an Ivy Bridge 3.3GHz E3-1230-V2 Xeon with 2 x 1TB SAS, RAID 1 for the first year and $1,188 USD/year afterwards.
While I'm happy so far with the QNAP, I do have a concern with how quickly it can rebuild RAID 6 with its miniscule Atom 525 processor should a drive fail.
My thought would be the following:
1) It is safer for a DR perspective to host the new server at home as I already have a dedicated full time VPN tunnel plus UPS all setup.
2) I can have the NAS built in to the new DR server and take advantage of the higher processing speed available with the server's CPU in comparison to any offerings from QNAP.
3) From costing perspective I surely must be able to buy hardware with the cost I'll be paying for just the first year of hosting, which is $1,687 USD.
I just went on Newegg and priced quickly the cost for sizing my own server with RAID built in. It works out to $1,623.92 USD for the following.
Intel Xeon E3-1230 V2 Ivy Bridge 3.3GHz (3.7GHz Turbo) LGA 1155 69W Quad-Core Server Processor BX80637E31230V2
ASUS P8B-X LGA 1155 Intel C202 ATX Intel Xeon E3 Server Motherboard
Kingston 8GB (2 x 4GB) 240-Pin DDR3 SDRAM DDR3 1333 (PC3 10600) ECC Unbuffered Server Memory Model KVR1333D3E9SK2/8G
Adaptec RAID 6405 2271100-R 6Gb/s SATA/SAS 4 internal ports w/512MB cache memory Controller Card, Kit
Four WD RE WD200MFYYG 2TB 7200 RPM 32MB Cache SAS 6Gb/s 3.5" Internal Enterprise Hard Drive Bare Drive
While this all sounds great I must admit I have very little knowledge on running RAID. I did encounter an experience with one of the servers offered by the same hosting company that has RAID 1 built in. One of the drives failed earlier and they attempted to swap in a new drive to have it rebuilt but there was an issue with the RAID card (not sure what the exact issue was) and they ended up having to have me to migrate everything on to a new server box as they had concerns that swapping in a new RAID card may corrupt data on the remaining drive. Not sure whether this was due to a poor training on their part on RAID rebuilt or this can indeed happen with RAID cards in servers.
I have also read on forums with people finding themselves with failed RAID cards but unable to swap in a new one due to different ways of storing bits and bytes of data on the disks by the old and new RAID cards. Not sure if this is still true or the new cards nowadays are standards compliant so will work no matter what card I swap in as long as it supports the same RAID level.
What I'm interested to hear is can anyone who have experience running RAID on servers be able to advise the disadvantages of storing data I need to protect on a server rather than on a dedicated NAS box such as the ones offered by QNAP? And what kind of things I need to be aware of? I'm fine with buying additional RAID cards and disks as backup so I can swap them in when they fail but I'm sure there are something I'm not aware of in terms of disadvantages or maintenance or whatever. Please comment. Thanks.
Are you intending to share this storage to multiple machines? If not there's no real reason to do NFS and plenty of reasons not to do so (performance being chief among them - internal storage is going to perform better than NFS presented storage).
I've used RAID1, RAID5, RAID6 and RAID10 (and of course JBOD). I've not seen the issues with RAID1 that you mention but of course it is always possible to have odd issues. Typically in a RAID1 situation you're using less storage anyway so don't lose as much if somehow you did lose both disks. RAID5 on the other hand is usually using at least 3 disks and often many more so you're apt to lose much more and it fails completely if you lose any 2 disks at the same time. RAID6 and RAID10 both can survive losing 2 disks at the same time. It isn't often that 2 fail at the same time (without the machine having exploded) but it does occur often enough to make RAID5 scary. I've yet to see 3 fail at the same time so have no qualms about running RAID6 or RAID10.
I'd heartily recommend internal RAID over any sort of NFS presented storage. However you should be aware that many folks think internal RAID controllers on Linux are actually slower than using Linux's built in mdadm (metadisk/software RAID). If cost is your driving factor then using software RAID might be a better idea. Additionally with metadisk you don't have to worry about getting a compatible RAID controller later on as there is no controller in the first place.
Here we've used Dell PowerEdge for years and they mostly have been purchased with the DELL PERC/SAS RAID controllers. Those are acctually OEM'ed from LSI by Dell so if you're looking at controllers I'd recommend them as they have a fair number of features including ways to recover RAID in some scenarios where you might otherwise not be able to do so. I've never much worried about using these controllers because we typically have vendor support so it is Dell's job to provide the replacements.
Isn't the NAS presented via NFS? Are you intending to share this storage to multiple machines?
I think with the current QNAP NAS I'm using CIFS via the SAMBA that the NAS also hosts. I see I'd have to install SAMBA also for the new DR server built with RAID in order to share it over the networks.
Just for reference sake I searched online to see how fast the SAS drives can transfer. 130 - 200MBps is equivalent to 1.04 - 1.6Gbps. Most off the shelf NAS solutions can do 100MBps which is equivalent to 0.8Gbps.
Typically in a RAID1 situation you're using less storage anyway so don't lose as much if somehow you did lose both disks.
I guess in this case you are referring to systems with RAID 1 where you have more than 2 disks mirrored. So say for example in a 3 disks RAID 1 setup you will still have one disk remain if you lose 2 disks.
I'd heartily recommend internal RAID over any sort of NFS presented storage.
Okay the RAID will be built in to the DR Server so the DR box can take advantage of the full local speed access it can.
However you should be aware that many folks think internal RAID controllers on Linux are actually slower than using Linux's built in mdadm (metadisk/software RAID). If cost is your driving factor then using software RAID might be a better idea. Additionally with metadisk you don't have to worry about getting a compatible RAID controller later on as there is no controller in the first place.
I just researched on the software RAID vs hardware RAID difference. Last I looked at RAID, mdadm was around but didn't have the same proliferation as it does today. And yes I can see why you said software RAID is a good contender. I think I can summarize the pros and cons to the following: Software RAID Pros
- Faster than fake hardware RAID.
- Incurs low CPU overhead and certainly I have confidence of a faster rebuild using a Core i7 with 32GB RAM than any Atom in a NAS, or probably even real hardware RAID cards for that matter.
- The ability to take the disks and put them into new motherboards without having to be concerned about which RAID card or manufacturer I need to stick to is a huge win as this was a concern I have been having. However need to do more research on this like whether there are any caveats and is it really as simple as taking the whole disk array out and swapping it into a new server box.
Software RAID Cons
- Doesn't offer the BBU or Flash Cache available on real RAID cards. Isn't it a potential issue if the application believes data has been written and the OS hasn't flushed it out to disk when some critical hardware (ie PSU, CPU, MB, etc) fails?
- ESXi doesn't support it. This is a big issue for me. A real big one. https://communities.vmware.com/thread/460033
How about SSD
Now the other alternative is to use SSD, inspired by "augmentedtrader" article below. Curiosity striked. Went on Newegg to price a 1TB SSD. While attractive it's still not cheap.
The problem is while I can run the ESXi on Hardware RAID 6, the RAID rebuild is still not going to take advantage of the host CPU (ie the Core i7 or Xeon) but rather the RAID card itself. And I'm seriously not too sure how much quicker will the on board processor is compared to an Atom.
Now this comes up to $2,043 and is only $356 more than $1,687 that I'd have to pay for another box to the hosting provider in just the first year. And the other plus side is it's free after that. I have both a dedicated DR box on ESXi RAID 1 with 2 disks failure redudancy and a dedicated software RAID 6 to take advantage of the processing power of Core i7. This is real attractive solution.
I guess I should have said "Isn't the NAS sharing out using NFS or SMB shares?". Either way the limiting factor is the network rather than the storage.
But it seems like you've gone on with further research based on my comments so already discovered what you need to be considering.
SSDs are definitely faster than mechanical drives by quite a bit. However, they do have other issues regarding bits that can't be written to or read later. Most of them have facilities for addressing this but you'd want to make sure the ones you choose do. Also you'd still want to do a RAID configuration for them because any SSD can fail.
We had the slot based SSD on a card one from FusionIO and they tell you they have onboard redundancy but leave out the fact that the firmware chip itself can be a victim to heat and cause the board to fail. Most of the reading I did after we saw that failure indicated most folks using the boards have them in a RAID1 configuration rather than relying on a single one.
Of course disk form factor SSDs rather than slot based are the way to go if you can afford it. We've been running SSDs in our big Hitachi array for 3 years now and have seen no issues from that. We've not yet done SSDs disk form factor within hosts - just the FusionIO stuff. (By the way we only had the issue with one FusionIO board so I don't want to give the impression they're bad - just that you DO have to be cognizant of where you put in in the system and keep track of the temperature.)
Wow you guys are using Fusion stuff. These products handle insane GBps speed (guess one day we'll see this type of speed on ultrabooks too). What are these servers being used for requiring these insane speed?
You mention disk form factors are more expensive but surely the Fusion stuff should be more, right?
I don't know that disk form factor would be any more or less expensive. As noted we've not used disk form factor at host level - just in the Hitachi disk array so we present storage via Fibre SAN using Qlogic HBAs int he servers.
For the most part we're using the FusionIO cards for common temporary table/data space for our larger database systems. This speeds up overall database performance without risking loss of data.
As I alluded to earlier we did at one point try to use one of the early cards as permanent storage but it wasn't installed redundantly (i.e. with another card) and we had a heat event that fried the firmware chip causing us a major outage. Fusion's idea of support doesn't seem to include the idea of 24/7 and immediate part replacement so we've been unwilling to use it for any permanent storage since then. As I noted after our event I found that most people that ARE using it for critical purposes buy them in pairs and put them in a RAID1 configuration so that one card failing isn't a single point of failure. It should also be noted that the failure I'm talking about was 2 years ago and hopefully they've made things better since then. The replacement we got back then is still in use to this day and we've since bought newer cards from them and none of these have failed but then again none of them are under quite the same load as they were when we tried to use that first one as permanent storage.
Raid defines hdd structures , NAS is network attached storage
Definitley yes , get some old cheap machine , put there some HDD's ,and stuff there all Samba / FTP / SSH users .
software RAID like mdadm will rebuild a faulty drive like 135 GB SAS in 45 minutes , over a 65 MB/s read/write speed.
SSD drive do that better, mine does 550 MB/s read /write ( 64 GB Corsair GT red label )
I did that with a humble raspberry pi ARM CPU and some usb drives, ...cheap , secure but brain storming is required.
Linux is fun , but some answers aint coming without intensive search and work involved
Another server of mine is a low amd xp 1800 since 2002 , 1,5 Ghz 512 ddr 400 Ram .. works like a charm !
Believe it or not , 80 % from internet is built on low spec machines