|
Build own RAID server rather than buying NAS solutions?
Hi guys I'm faced with a dilemma of whether to build my own RAID and need some advise and experience comments.
Current situation is I already have a QNAP TS-459 Pro+ NAS with RAID 6 configured. And I will have the need to add a new server to function as DR. If I decide to co-locate my new box to be in the same data center as my existing box, the hosting provider will charge $1,687 USD for an Ivy Bridge 3.3GHz E3-1230-V2 Xeon with 2 x 1TB SAS, RAID 1 for the first year and $1,188 USD/year afterwards. While I'm happy so far with the QNAP, I do have a concern with how quickly it can rebuild RAID 6 with its miniscule Atom 525 processor should a drive fail. My thought would be the following: 1) It is safer for a DR perspective to host the new server at home as I already have a dedicated full time VPN tunnel plus UPS all setup. 2) I can have the NAS built in to the new DR server and take advantage of the higher processing speed available with the server's CPU in comparison to any offerings from QNAP. 3) From costing perspective I surely must be able to buy hardware with the cost I'll be paying for just the first year of hosting, which is $1,687 USD. I just went on Newegg and priced quickly the cost for sizing my own server with RAID built in. It works out to $1,623.92 USD for the following. Intel Xeon E3-1230 V2 Ivy Bridge 3.3GHz (3.7GHz Turbo) LGA 1155 69W Quad-Core Server Processor BX80637E31230V2 ASUS P8B-X LGA 1155 Intel C202 ATX Intel Xeon E3 Server Motherboard Kingston 8GB (2 x 4GB) 240-Pin DDR3 SDRAM DDR3 1333 (PC3 10600) ECC Unbuffered Server Memory Model KVR1333D3E9SK2/8G Adaptec RAID 6405 2271100-R 6Gb/s SATA/SAS 4 internal ports w/512MB cache memory Controller Card, Kit Four WD RE WD200MFYYG 2TB 7200 RPM 32MB Cache SAS 6Gb/s 3.5" Internal Enterprise Hard Drive Bare Drive While this all sounds great I must admit I have very little knowledge on running RAID. I did encounter an experience with one of the servers offered by the same hosting company that has RAID 1 built in. One of the drives failed earlier and they attempted to swap in a new drive to have it rebuilt but there was an issue with the RAID card (not sure what the exact issue was) and they ended up having to have me to migrate everything on to a new server box as they had concerns that swapping in a new RAID card may corrupt data on the remaining drive. Not sure whether this was due to a poor training on their part on RAID rebuilt or this can indeed happen with RAID cards in servers. I have also read on forums with people finding themselves with failed RAID cards but unable to swap in a new one due to different ways of storing bits and bytes of data on the disks by the old and new RAID cards. Not sure if this is still true or the new cards nowadays are standards compliant so will work no matter what card I swap in as long as it supports the same RAID level. What I'm interested to hear is can anyone who have experience running RAID on servers be able to advise the disadvantages of storing data I need to protect on a server rather than on a dedicated NAS box such as the ones offered by QNAP? And what kind of things I need to be aware of? I'm fine with buying additional RAID cards and disks as backup so I can swap them in when they fail but I'm sure there are something I'm not aware of in terms of disadvantages or maintenance or whatever. Please comment. Thanks. |
Isn't the NAS presented via NFS?
Are you intending to share this storage to multiple machines? If not there's no real reason to do NFS and plenty of reasons not to do so (performance being chief among them - internal storage is going to perform better than NFS presented storage). I've used RAID1, RAID5, RAID6 and RAID10 (and of course JBOD). I've not seen the issues with RAID1 that you mention but of course it is always possible to have odd issues. Typically in a RAID1 situation you're using less storage anyway so don't lose as much if somehow you did lose both disks. RAID5 on the other hand is usually using at least 3 disks and often many more so you're apt to lose much more and it fails completely if you lose any 2 disks at the same time. RAID6 and RAID10 both can survive losing 2 disks at the same time. It isn't often that 2 fail at the same time (without the machine having exploded) but it does occur often enough to make RAID5 scary. I've yet to see 3 fail at the same time so have no qualms about running RAID6 or RAID10. I'd heartily recommend internal RAID over any sort of NFS presented storage. However you should be aware that many folks think internal RAID controllers on Linux are actually slower than using Linux's built in mdadm (metadisk/software RAID). If cost is your driving factor then using software RAID might be a better idea. Additionally with metadisk you don't have to worry about getting a compatible RAID controller later on as there is no controller in the first place. Here we've used Dell PowerEdge for years and they mostly have been purchased with the DELL PERC/SAS RAID controllers. Those are acctually OEM'ed from LSI by Dell so if you're looking at controllers I'd recommend them as they have a fair number of features including ways to recover RAID in some scenarios where you might otherwise not be able to do so. I've never much worried about using these controllers because we typically have vendor support so it is Dell's job to provide the replacements. |
Quote:
Just for reference sake I searched online to see how fast the SAS drives can transfer. 130 - 200MBps is equivalent to 1.04 - 1.6Gbps. Most off the shelf NAS solutions can do 100MBps which is equivalent to 0.8Gbps. 2012 SAS Benchmark (130 - 200MBps for 2MB based sequential read/write) http://www.storagereview.com/toshiba...d_drive_review 2007 SAS Benchmark (350MBps for RAID 0 read/write) http://www.tomshardware.com/reviews/...es,1702-6.html 350MBps / 2 = 175MBps per drive so number falls in the 130 - 200MBps figure from above. Quote:
Quote:
Quote:
Software RAID Pros - Faster than fake hardware RAID. - Incurs low CPU overhead and certainly I have confidence of a faster rebuild using a Core i7 with 32GB RAM than any Atom in a NAS, or probably even real hardware RAID cards for that matter. - The ability to take the disks and put them into new motherboards without having to be concerned about which RAID card or manufacturer I need to stick to is a huge win as this was a concern I have been having. However need to do more research on this like whether there are any caveats and is it really as simple as taking the whole disk array out and swapping it into a new server box. Software RAID Cons - Doesn't offer the BBU or Flash Cache available on real RAID cards. Isn't it a potential issue if the application believes data has been written and the OS hasn't flushed it out to disk when some critical hardware (ie PSU, CPU, MB, etc) fails? - ESXi doesn't support it. This is a big issue for me. A real big one. https://communities.vmware.com/thread/460033 How about SSD Now the other alternative is to use SSD, inspired by "augmentedtrader" article below. Curiosity striked. Went on Newegg to price a 1TB SSD. While attractive it's still not cheap. Single Disk 1TB SSD Solution - Superb performance to cost ratio but still run the risk of failing the only disk we have http://www.newegg.com/Product/Produc...82E16820147251 $569 SAMSUNG 840 EVO MZ-7TE1T0BW 2.5" 1TB SATA III TLC Internal Solid State Drive (SSD) Hardware RAID 1 1TB SSD Solution - If we have concerns with running only one SSD and it fails http://www.newegg.com/Product/Produc...82E16816103102 http://www.newegg.com/Product/Produc...82E16820147251 $194 Adaptec RAID 2405 2260200-R SATA/SAS 4 internal ports w/ 128MB cache memory Controller Card, Single $569 x 2 SAMSUNG 840 EVO MZ-7TE1T0BW 2.5" 1TB SATA III TLC Internal Solid State Drive (SSD) $1,332 Hardware RAID 6 1TB Spinning Disk Solution - Low cost solution with 2 disks failure redundancy http://www.newegg.com/Product/Produc...82E16816103097 http://www.newegg.com/Product/Produc...82E16822136497 $332 Adaptec RAID 5405 2258200-R SATA/SAS 4 internal ports w/ 256MB cache memory Controller Card, Single $63 x 4 Western Digital WD AV-GP WD5000AVDS 500GB 32MB Cache SATA 3.0Gb/s 3.5" Internal AV Hard Drive Bare Drive $584 NAS RAID 6 1TB Spinning Disk Solution http://www.newegg.com/Product/Produc...82E16822107084 http://www.newegg.com/Product/Produc...82E16822136497 $750 QNAP TS-469-PRO-US Diskless System 4-Bay, All-in-One NAS $63 x 4 Western Digital WD AV-GP WD5000AVDS 500GB 32MB Cache SATA 3.0Gb/s 3.5" Internal AV Hard Drive Bare Drive $1,002 The problem The problem is while I can run the ESXi on Hardware RAID 6, the RAID rebuild is still not going to take advantage of the host CPU (ie the Core i7 or Xeon) but rather the RAID card itself. And I'm seriously not too sure how much quicker will the on board processor is compared to an Atom. The solution In summary it appears the most logical solution is as follow. Note the Shuttle SH67H3 is Ivy Bridge / Sandy Bridge compatible only so won't work with the Haswell yet. Have a look The Perfect VMware vSphere 5 Home Lab and Shuttle SZ87R6 VMware ESXi 5.5 Home Lab. ESXi DR Server $200 Shuttle SH67H3 $300 Intel Core i7-3770 Ivy Bridge 3.4GHz (3.9GHz Turbo) LGA 1155 77W Quad-Core Desktop Processor Intel HD Graphics 4000 BX80637I73770 $204 Kingston 16GB (2 x 8GB) 240-Pin DDR3 SDRAM ECC Unbuffered DDR3 1333 Server Memory Model KVR1333D3E9SK2/16G $194 Adaptec RAID 2405 2260200-R SATA/SAS 4 internal ports w/ 128MB cache memory Controller Card, Single $63 x 3 Western Digital WD AV-GP WD5000AVDS 500GB 32MB Cache SATA 3.0Gb/s 3.5" Internal AV Hard Drive Bare Drive $1,087 Home Brew mdadm Software RAID 6 - Note I don't think I can fit 4 SATA drives into the Shuttle but the pricing should be indicative of what I'd have to pay for an alternative solution. $200 Shuttle SH67H3 $300 Intel Core i7-3770 Ivy Bridge 3.4GHz (3.9GHz Turbo) LGA 1155 77W Quad-Core Desktop Processor Intel HD Graphics 4000 BX80637I73770 $204 Kingston 16GB (2 x 8GB) 240-Pin DDR3 SDRAM ECC Unbuffered DDR3 1333 Server Memory Model KVR1333D3E9SK2/16G $63 x 4 Western Digital WD AV-GP WD5000AVDS 500GB 32MB Cache SATA 3.0Gb/s 3.5" Internal AV Hard Drive Bare Drive $956 Now this comes up to $2,043 and is only $356 more than $1,687 that I'd have to pay for another box to the hosting provider in just the first year. And the other plus side is it's free after that. I have both a dedicated DR box on ESXi RAID 1 with 2 disks failure redudancy and a dedicated software RAID 6 to take advantage of the processing power of Core i7. This is real attractive solution. Some readings for reference Generic Software RAID vs Hardware RAID http://augmentedtrader.wordpress.com...0-things-raid/ http://blog.ioflood.com/?p=170 http://www.linux.com/news/hardware/s...-software-raid http://skrypuch.com/raid/ Issues experienced with Hardware RAID http://serverfault.com/questions/539...-rebuild-array http://blog.dimensionzero.ca/?p=399 |
I guess I should have said "Isn't the NAS sharing out using NFS or SMB shares?". Either way the limiting factor is the network rather than the storage.
But it seems like you've gone on with further research based on my comments so already discovered what you need to be considering. SSDs are definitely faster than mechanical drives by quite a bit. However, they do have other issues regarding bits that can't be written to or read later. Most of them have facilities for addressing this but you'd want to make sure the ones you choose do. Also you'd still want to do a RAID configuration for them because any SSD can fail. We had the slot based SSD on a card one from FusionIO and they tell you they have onboard redundancy but leave out the fact that the firmware chip itself can be a victim to heat and cause the board to fail. Most of the reading I did after we saw that failure indicated most folks using the boards have them in a RAID1 configuration rather than relying on a single one. Of course disk form factor SSDs rather than slot based are the way to go if you can afford it. We've been running SSDs in our big Hitachi array for 3 years now and have seen no issues from that. We've not yet done SSDs disk form factor within hosts - just the FusionIO stuff. (By the way we only had the issue with one FusionIO board so I don't want to give the impression they're bad - just that you DO have to be cognizant of where you put in in the system and keep track of the temperature.) |
Wow you guys are using Fusion stuff. These products handle insane GBps speed (guess one day we'll see this type of speed on ultrabooks too). What are these servers being used for requiring these insane speed?
You mention disk form factors are more expensive but surely the Fusion stuff should be more, right? |
I don't know that disk form factor would be any more or less expensive. As noted we've not used disk form factor at host level - just in the Hitachi disk array so we present storage via Fibre SAN using Qlogic HBAs int he servers.
For the most part we're using the FusionIO cards for common temporary table/data space for our larger database systems. This speeds up overall database performance without risking loss of data. As I alluded to earlier we did at one point try to use one of the early cards as permanent storage but it wasn't installed redundantly (i.e. with another card) and we had a heat event that fried the firmware chip causing us a major outage. Fusion's idea of support doesn't seem to include the idea of 24/7 and immediate part replacement so we've been unwilling to use it for any permanent storage since then. As I noted after our event I found that most people that ARE using it for critical purposes buy them in pairs and put them in a RAID1 configuration so that one card failing isn't a single point of failure. It should also be noted that the failure I'm talking about was 2 years ago and hopefully they've made things better since then. The replacement we got back then is still in use to this day and we've since bought newer cards from them and none of these have failed but then again none of them are under quite the same load as they were when we tried to use that first one as permanent storage. |
NAS vs RAID
Raid defines hdd structures , NAS is network attached storage
Definitley yes , get some old cheap machine , put there some HDD's ,and stuff there all Samba / FTP / SSH users . software RAID like mdadm will rebuild a faulty drive like 135 GB SAS in 45 minutes , over a 65 MB/s read/write speed. SSD drive do that better, mine does 550 MB/s read /write ( 64 GB Corsair GT red label ) I did that with a humble raspberry pi ARM CPU and some usb drives, ...cheap , secure but brain storming is required. Linux is fun , but some answers aint coming without intensive search and work involved Another server of mine is a low amd xp 1800 since 2002 , 1,5 Ghz 512 ddr 400 Ram .. works like a charm ! Believe it or not , 80 % from internet is built on low spec machines |
The atom shouldn't affect how fast the drive gets rebuilt for the most part. That nas is pretty good. How many times a year do you change a drive in it? A quality nas drive ought to last years.
|
Quote:
I guess it's either limited by CPU calculating parity or disk IO. Have there been any testing done to see which one is the real limiting factor? |
The longest "rebuild" I've seen is on 2 TB SATA 7.2k drives but even those don't take days.
We have some NAS devices (including Deduplication devices presented as NAS) and even on the largest drives (1 TB) I've seen on those rebuild didn't take more than a couple of hours. |
Quote:
|
I can't really say in either case.
For the Hitachi it is a dedicated storage unit that has its own design and algorithms including a sizeable system cache so isn't comparable to the rest. For the NAS devices I've never really checked to see if they're using controllers or software RAID. For the systems we have that are using internal storage they all have the PERC/SAS (LSI OEM) RAID controllers I previously mentioned. My point was drive rebuilds themselves don't take that long due to the drives. A previous poster was suggesting that the controller you're speaking of wouldn't be a limiting factor. |
Interesting I just realized there's a speed limit imposed on Linux RAID with following!
/proc/sys/dev/raid/speed_limit_min Have a look at the following: http://forum.qnap.com/viewtopic.php?t=10268 http://forum.qnap.com/viewtopic.php?t=10268 http://www.cyberciti.biz/tips/linux-...ild-speed.html |
| All times are GMT -5. The time now is 08:07 AM. |