LinuxQuestions.org
Visit Jeremy's Blog.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Server
User Name
Password
Linux - Server This forum is for the discussion of Linux Software used in a server related context.

Notices


Reply
  Search this Thread
Old 12-26-2013, 07:46 PM   #1
grob115
Member
 
Registered: Oct 2005
Posts: 540

Rep: Reputation: 32
Build own RAID server rather than buying NAS solutions?


Hi guys I'm faced with a dilemma of whether to build my own RAID and need some advise and experience comments.

Current situation is I already have a QNAP TS-459 Pro+ NAS with RAID 6 configured. And I will have the need to add a new server to function as DR. If I decide to co-locate my new box to be in the same data center as my existing box, the hosting provider will charge $1,687 USD for an Ivy Bridge 3.3GHz E3-1230-V2 Xeon with 2 x 1TB SAS, RAID 1 for the first year and $1,188 USD/year afterwards.

While I'm happy so far with the QNAP, I do have a concern with how quickly it can rebuild RAID 6 with its miniscule Atom 525 processor should a drive fail.

My thought would be the following:
1) It is safer for a DR perspective to host the new server at home as I already have a dedicated full time VPN tunnel plus UPS all setup.
2) I can have the NAS built in to the new DR server and take advantage of the higher processing speed available with the server's CPU in comparison to any offerings from QNAP.
3) From costing perspective I surely must be able to buy hardware with the cost I'll be paying for just the first year of hosting, which is $1,687 USD.

I just went on Newegg and priced quickly the cost for sizing my own server with RAID built in. It works out to $1,623.92 USD for the following.
Intel Xeon E3-1230 V2 Ivy Bridge 3.3GHz (3.7GHz Turbo) LGA 1155 69W Quad-Core Server Processor BX80637E31230V2
ASUS P8B-X LGA 1155 Intel C202 ATX Intel Xeon E3 Server Motherboard
Kingston 8GB (2 x 4GB) 240-Pin DDR3 SDRAM DDR3 1333 (PC3 10600) ECC Unbuffered Server Memory Model KVR1333D3E9SK2/8G
Adaptec RAID 6405 2271100-R 6Gb/s SATA/SAS 4 internal ports w/512MB cache memory Controller Card, Kit
Four WD RE WD200MFYYG 2TB 7200 RPM 32MB Cache SAS 6Gb/s 3.5" Internal Enterprise Hard Drive Bare Drive

While this all sounds great I must admit I have very little knowledge on running RAID. I did encounter an experience with one of the servers offered by the same hosting company that has RAID 1 built in. One of the drives failed earlier and they attempted to swap in a new drive to have it rebuilt but there was an issue with the RAID card (not sure what the exact issue was) and they ended up having to have me to migrate everything on to a new server box as they had concerns that swapping in a new RAID card may corrupt data on the remaining drive. Not sure whether this was due to a poor training on their part on RAID rebuilt or this can indeed happen with RAID cards in servers.

I have also read on forums with people finding themselves with failed RAID cards but unable to swap in a new one due to different ways of storing bits and bytes of data on the disks by the old and new RAID cards. Not sure if this is still true or the new cards nowadays are standards compliant so will work no matter what card I swap in as long as it supports the same RAID level.

What I'm interested to hear is can anyone who have experience running RAID on servers be able to advise the disadvantages of storing data I need to protect on a server rather than on a dedicated NAS box such as the ones offered by QNAP? And what kind of things I need to be aware of? I'm fine with buying additional RAID cards and disks as backup so I can swap them in when they fail but I'm sure there are something I'm not aware of in terms of disadvantages or maintenance or whatever. Please comment. Thanks.
 
Old 12-27-2013, 03:26 PM   #2
MensaWater
LQ Guru
 
Registered: May 2005
Location: Atlanta Georgia USA
Distribution: Redhat (RHEL), CentOS, Fedora, CoreOS, Debian, FreeBSD, HP-UX, Solaris, SCO
Posts: 6,954
Blog Entries: 14

Rep: Reputation: 1175Reputation: 1175Reputation: 1175Reputation: 1175Reputation: 1175Reputation: 1175Reputation: 1175Reputation: 1175Reputation: 1175
Isn't the NAS presented via NFS?

Are you intending to share this storage to multiple machines? If not there's no real reason to do NFS and plenty of reasons not to do so (performance being chief among them - internal storage is going to perform better than NFS presented storage).

I've used RAID1, RAID5, RAID6 and RAID10 (and of course JBOD). I've not seen the issues with RAID1 that you mention but of course it is always possible to have odd issues. Typically in a RAID1 situation you're using less storage anyway so don't lose as much if somehow you did lose both disks. RAID5 on the other hand is usually using at least 3 disks and often many more so you're apt to lose much more and it fails completely if you lose any 2 disks at the same time. RAID6 and RAID10 both can survive losing 2 disks at the same time. It isn't often that 2 fail at the same time (without the machine having exploded) but it does occur often enough to make RAID5 scary. I've yet to see 3 fail at the same time so have no qualms about running RAID6 or RAID10.

I'd heartily recommend internal RAID over any sort of NFS presented storage. However you should be aware that many folks think internal RAID controllers on Linux are actually slower than using Linux's built in mdadm (metadisk/software RAID). If cost is your driving factor then using software RAID might be a better idea. Additionally with metadisk you don't have to worry about getting a compatible RAID controller later on as there is no controller in the first place.

Here we've used Dell PowerEdge for years and they mostly have been purchased with the DELL PERC/SAS RAID controllers. Those are acctually OEM'ed from LSI by Dell so if you're looking at controllers I'd recommend them as they have a fair number of features including ways to recover RAID in some scenarios where you might otherwise not be able to do so. I've never much worried about using these controllers because we typically have vendor support so it is Dell's job to provide the replacements.
 
Old 12-28-2013, 06:02 AM   #3
grob115
Member
 
Registered: Oct 2005
Posts: 540

Original Poster
Rep: Reputation: 32
Quote:
Isn't the NAS presented via NFS? Are you intending to share this storage to multiple machines?
I think with the current QNAP NAS I'm using CIFS via the SAMBA that the NAS also hosts. I see I'd have to install SAMBA also for the new DR server built with RAID in order to share it over the networks.

Just for reference sake I searched online to see how fast the SAS drives can transfer. 130 - 200MBps is equivalent to 1.04 - 1.6Gbps. Most off the shelf NAS solutions can do 100MBps which is equivalent to 0.8Gbps.

2012 SAS Benchmark (130 - 200MBps for 2MB based sequential read/write)
http://www.storagereview.com/toshiba...d_drive_review

2007 SAS Benchmark (350MBps for RAID 0 read/write)
http://www.tomshardware.com/reviews/...es,1702-6.html
350MBps / 2 = 175MBps per drive so number falls in the 130 - 200MBps figure from above.


Quote:
Typically in a RAID1 situation you're using less storage anyway so don't lose as much if somehow you did lose both disks.
I guess in this case you are referring to systems with RAID 1 where you have more than 2 disks mirrored. So say for example in a 3 disks RAID 1 setup you will still have one disk remain if you lose 2 disks.


Quote:
I'd heartily recommend internal RAID over any sort of NFS presented storage.
Okay the RAID will be built in to the DR Server so the DR box can take advantage of the full local speed access it can.


Quote:
However you should be aware that many folks think internal RAID controllers on Linux are actually slower than using Linux's built in mdadm (metadisk/software RAID). If cost is your driving factor then using software RAID might be a better idea. Additionally with metadisk you don't have to worry about getting a compatible RAID controller later on as there is no controller in the first place.
I just researched on the software RAID vs hardware RAID difference. Last I looked at RAID, mdadm was around but didn't have the same proliferation as it does today. And yes I can see why you said software RAID is a good contender. I think I can summarize the pros and cons to the following:
Software RAID Pros
- Faster than fake hardware RAID.
- Incurs low CPU overhead and certainly I have confidence of a faster rebuild using a Core i7 with 32GB RAM than any Atom in a NAS, or probably even real hardware RAID cards for that matter.
- The ability to take the disks and put them into new motherboards without having to be concerned about which RAID card or manufacturer I need to stick to is a huge win as this was a concern I have been having. However need to do more research on this like whether there are any caveats and is it really as simple as taking the whole disk array out and swapping it into a new server box.

Software RAID Cons
- Doesn't offer the BBU or Flash Cache available on real RAID cards. Isn't it a potential issue if the application believes data has been written and the OS hasn't flushed it out to disk when some critical hardware (ie PSU, CPU, MB, etc) fails?
- ESXi doesn't support it. This is a big issue for me. A real big one.
https://communities.vmware.com/thread/460033


How about SSD
Now the other alternative is to use SSD, inspired by "augmentedtrader" article below. Curiosity striked. Went on Newegg to price a 1TB SSD. While attractive it's still not cheap.

Single Disk 1TB SSD Solution
- Superb performance to cost ratio but still run the risk of failing the only disk we have
http://www.newegg.com/Product/Produc...82E16820147251
$569 SAMSUNG 840 EVO MZ-7TE1T0BW 2.5" 1TB SATA III TLC Internal Solid State Drive (SSD)

Hardware RAID 1 1TB SSD Solution
- If we have concerns with running only one SSD and it fails
http://www.newegg.com/Product/Produc...82E16816103102
http://www.newegg.com/Product/Produc...82E16820147251
$194 Adaptec RAID 2405 2260200-R SATA/SAS 4 internal ports w/ 128MB cache memory Controller Card, Single
$569 x 2 SAMSUNG 840 EVO MZ-7TE1T0BW 2.5" 1TB SATA III TLC Internal Solid State Drive (SSD)
$1,332

Hardware RAID 6 1TB Spinning Disk Solution
- Low cost solution with 2 disks failure redundancy
http://www.newegg.com/Product/Produc...82E16816103097
http://www.newegg.com/Product/Produc...82E16822136497
$332 Adaptec RAID 5405 2258200-R SATA/SAS 4 internal ports w/ 256MB cache memory Controller Card, Single
$63 x 4 Western Digital WD AV-GP WD5000AVDS 500GB 32MB Cache SATA 3.0Gb/s 3.5" Internal AV Hard Drive Bare Drive
$584

NAS RAID 6 1TB Spinning Disk Solution
http://www.newegg.com/Product/Produc...82E16822107084
http://www.newegg.com/Product/Produc...82E16822136497
$750 QNAP TS-469-PRO-US Diskless System 4-Bay, All-in-One NAS
$63 x 4 Western Digital WD AV-GP WD5000AVDS 500GB 32MB Cache SATA 3.0Gb/s 3.5" Internal AV Hard Drive Bare Drive
$1,002


The problem
The problem is while I can run the ESXi on Hardware RAID 6, the RAID rebuild is still not going to take advantage of the host CPU (ie the Core i7 or Xeon) but rather the RAID card itself. And I'm seriously not too sure how much quicker will the on board processor is compared to an Atom.


The solution
In summary it appears the most logical solution is as follow. Note the Shuttle SH67H3 is Ivy Bridge / Sandy Bridge compatible only so won't work with the Haswell yet. Have a look The Perfect VMware vSphere 5 Home Lab and Shuttle SZ87R6 VMware ESXi 5.5 Home Lab.

ESXi DR Server
$200 Shuttle SH67H3
$300 Intel Core i7-3770 Ivy Bridge 3.4GHz (3.9GHz Turbo) LGA 1155 77W Quad-Core Desktop Processor Intel HD Graphics 4000 BX80637I73770
$204 Kingston 16GB (2 x 8GB) 240-Pin DDR3 SDRAM ECC Unbuffered DDR3 1333 Server Memory Model KVR1333D3E9SK2/16G
$194 Adaptec RAID 2405 2260200-R SATA/SAS 4 internal ports w/ 128MB cache memory Controller Card, Single
$63 x 3 Western Digital WD AV-GP WD5000AVDS 500GB 32MB Cache SATA 3.0Gb/s 3.5" Internal AV Hard Drive Bare Drive
$1,087

Home Brew mdadm Software RAID 6
- Note I don't think I can fit 4 SATA drives into the Shuttle but the pricing should be indicative of what I'd have to pay for an alternative solution.
$200 Shuttle SH67H3
$300 Intel Core i7-3770 Ivy Bridge 3.4GHz (3.9GHz Turbo) LGA 1155 77W Quad-Core Desktop Processor Intel HD Graphics 4000 BX80637I73770
$204 Kingston 16GB (2 x 8GB) 240-Pin DDR3 SDRAM ECC Unbuffered DDR3 1333 Server Memory Model KVR1333D3E9SK2/16G
$63 x 4 Western Digital WD AV-GP WD5000AVDS 500GB 32MB Cache SATA 3.0Gb/s 3.5" Internal AV Hard Drive Bare Drive
$956

Now this comes up to $2,043 and is only $356 more than $1,687 that I'd have to pay for another box to the hosting provider in just the first year. And the other plus side is it's free after that. I have both a dedicated DR box on ESXi RAID 1 with 2 disks failure redudancy and a dedicated software RAID 6 to take advantage of the processing power of Core i7. This is real attractive solution.


Some readings for reference
Generic Software RAID vs Hardware RAID
http://augmentedtrader.wordpress.com...0-things-raid/
http://blog.ioflood.com/?p=170
http://www.linux.com/news/hardware/s...-software-raid
http://skrypuch.com/raid/

Issues experienced with Hardware RAID
http://serverfault.com/questions/539...-rebuild-array
http://blog.dimensionzero.ca/?p=399

Last edited by grob115; 12-28-2013 at 07:14 PM.
 
Old 12-30-2013, 09:47 AM   #4
MensaWater
LQ Guru
 
Registered: May 2005
Location: Atlanta Georgia USA
Distribution: Redhat (RHEL), CentOS, Fedora, CoreOS, Debian, FreeBSD, HP-UX, Solaris, SCO
Posts: 6,954
Blog Entries: 14

Rep: Reputation: 1175Reputation: 1175Reputation: 1175Reputation: 1175Reputation: 1175Reputation: 1175Reputation: 1175Reputation: 1175Reputation: 1175
I guess I should have said "Isn't the NAS sharing out using NFS or SMB shares?". Either way the limiting factor is the network rather than the storage.

But it seems like you've gone on with further research based on my comments so already discovered what you need to be considering.

SSDs are definitely faster than mechanical drives by quite a bit. However, they do have other issues regarding bits that can't be written to or read later. Most of them have facilities for addressing this but you'd want to make sure the ones you choose do. Also you'd still want to do a RAID configuration for them because any SSD can fail.

We had the slot based SSD on a card one from FusionIO and they tell you they have onboard redundancy but leave out the fact that the firmware chip itself can be a victim to heat and cause the board to fail. Most of the reading I did after we saw that failure indicated most folks using the boards have them in a RAID1 configuration rather than relying on a single one.

Of course disk form factor SSDs rather than slot based are the way to go if you can afford it. We've been running SSDs in our big Hitachi array for 3 years now and have seen no issues from that. We've not yet done SSDs disk form factor within hosts - just the FusionIO stuff. (By the way we only had the issue with one FusionIO board so I don't want to give the impression they're bad - just that you DO have to be cognizant of where you put in in the system and keep track of the temperature.)
 
Old 12-31-2013, 02:25 AM   #5
grob115
Member
 
Registered: Oct 2005
Posts: 540

Original Poster
Rep: Reputation: 32
Wow you guys are using Fusion stuff. These products handle insane GBps speed (guess one day we'll see this type of speed on ultrabooks too). What are these servers being used for requiring these insane speed?

You mention disk form factors are more expensive but surely the Fusion stuff should be more, right?
 
Old 12-31-2013, 08:32 AM   #6
MensaWater
LQ Guru
 
Registered: May 2005
Location: Atlanta Georgia USA
Distribution: Redhat (RHEL), CentOS, Fedora, CoreOS, Debian, FreeBSD, HP-UX, Solaris, SCO
Posts: 6,954
Blog Entries: 14

Rep: Reputation: 1175Reputation: 1175Reputation: 1175Reputation: 1175Reputation: 1175Reputation: 1175Reputation: 1175Reputation: 1175Reputation: 1175
I don't know that disk form factor would be any more or less expensive. As noted we've not used disk form factor at host level - just in the Hitachi disk array so we present storage via Fibre SAN using Qlogic HBAs int he servers.

For the most part we're using the FusionIO cards for common temporary table/data space for our larger database systems. This speeds up overall database performance without risking loss of data.

As I alluded to earlier we did at one point try to use one of the early cards as permanent storage but it wasn't installed redundantly (i.e. with another card) and we had a heat event that fried the firmware chip causing us a major outage. Fusion's idea of support doesn't seem to include the idea of 24/7 and immediate part replacement so we've been unwilling to use it for any permanent storage since then. As I noted after our event I found that most people that ARE using it for critical purposes buy them in pairs and put them in a RAID1 configuration so that one card failing isn't a single point of failure. It should also be noted that the failure I'm talking about was 2 years ago and hopefully they've made things better since then. The replacement we got back then is still in use to this day and we've since bought newer cards from them and none of these have failed but then again none of them are under quite the same load as they were when we tried to use that first one as permanent storage.
 
Old 12-31-2013, 09:15 AM   #7
yo8rxp
Member
 
Registered: Jul 2009
Location: Romania
Distribution: Ubuntu 10.04 Gnome 2
Posts: 102

Rep: Reputation: 30
NAS vs RAID

Raid defines hdd structures , NAS is network attached storage
Definitley yes , get some old cheap machine , put there some HDD's ,and stuff there all Samba / FTP / SSH users .
software RAID like mdadm will rebuild a faulty drive like 135 GB SAS in 45 minutes , over a 65 MB/s read/write speed.
SSD drive do that better, mine does 550 MB/s read /write ( 64 GB Corsair GT red label )
I did that with a humble raspberry pi ARM CPU and some usb drives, ...cheap , secure but brain storming is required.

Linux is fun , but some answers aint coming without intensive search and work involved

Another server of mine is a low amd xp 1800 since 2002 , 1,5 Ghz 512 ddr 400 Ram .. works like a charm !
Believe it or not , 80 % from internet is built on low spec machines

Last edited by yo8rxp; 12-31-2013 at 09:19 AM.
 
Old 12-31-2013, 09:43 AM   #8
jefro
Moderator
 
Registered: Mar 2008
Posts: 16,889

Rep: Reputation: 2499Reputation: 2499Reputation: 2499Reputation: 2499Reputation: 2499Reputation: 2499Reputation: 2499Reputation: 2499Reputation: 2499Reputation: 2499Reputation: 2499
The atom shouldn't affect how fast the drive gets rebuilt for the most part. That nas is pretty good. How many times a year do you change a drive in it? A quality nas drive ought to last years.
 
Old 12-31-2013, 10:41 AM   #9
grob115
Member
 
Registered: Oct 2005
Posts: 540

Original Poster
Rep: Reputation: 32
Quote:
Originally Posted by jefro View Post
The atom shouldn't affect how fast the drive gets rebuilt for the most part. That nas is pretty good. How many times a year do you change a drive in it? A quality nas drive ought to last years.
Um... I thought part of the reason why RAID6 takes so long to rebuild (some reported days) is because most NAS are underpowered. No?

I guess it's either limited by CPU calculating parity or disk IO. Have there been any testing done to see which one is the real limiting factor?
 
Old 12-31-2013, 11:41 AM   #10
MensaWater
LQ Guru
 
Registered: May 2005
Location: Atlanta Georgia USA
Distribution: Redhat (RHEL), CentOS, Fedora, CoreOS, Debian, FreeBSD, HP-UX, Solaris, SCO
Posts: 6,954
Blog Entries: 14

Rep: Reputation: 1175Reputation: 1175Reputation: 1175Reputation: 1175Reputation: 1175Reputation: 1175Reputation: 1175Reputation: 1175Reputation: 1175
The longest "rebuild" I've seen is on 2 TB SATA 7.2k drives but even those don't take days.

We have some NAS devices (including Deduplication devices presented as NAS) and even on the largest drives (1 TB) I've seen on those rebuild didn't take more than a couple of hours.
 
Old 01-01-2014, 11:47 AM   #11
grob115
Member
 
Registered: Oct 2005
Posts: 540

Original Poster
Rep: Reputation: 32
Quote:
Originally Posted by MensaWater View Post
The longest "rebuild" I've seen is on 2 TB SATA 7.2k drives but even those don't take days.

We have some NAS devices (including Deduplication devices presented as NAS) and even on the largest drives (1 TB) I've seen on those rebuild didn't take more than a couple of hours.
Sorry were these based on software RAID on proper hardware like with a Core i5 or Core i7? Or with processors like Atom D525 that comes with products like QNAP?
 
Old 01-02-2014, 09:06 AM   #12
MensaWater
LQ Guru
 
Registered: May 2005
Location: Atlanta Georgia USA
Distribution: Redhat (RHEL), CentOS, Fedora, CoreOS, Debian, FreeBSD, HP-UX, Solaris, SCO
Posts: 6,954
Blog Entries: 14

Rep: Reputation: 1175Reputation: 1175Reputation: 1175Reputation: 1175Reputation: 1175Reputation: 1175Reputation: 1175Reputation: 1175Reputation: 1175
I can't really say in either case.

For the Hitachi it is a dedicated storage unit that has its own design and algorithms including a sizeable system cache so isn't comparable to the rest.

For the NAS devices I've never really checked to see if they're using controllers or software RAID.

For the systems we have that are using internal storage they all have the PERC/SAS (LSI OEM) RAID controllers I previously mentioned.

My point was drive rebuilds themselves don't take that long due to the drives.

A previous poster was suggesting that the controller you're speaking of wouldn't be a limiting factor.
 
Old 01-02-2014, 09:38 AM   #13
grob115
Member
 
Registered: Oct 2005
Posts: 540

Original Poster
Rep: Reputation: 32
Interesting I just realized there's a speed limit imposed on Linux RAID with following!
/proc/sys/dev/raid/speed_limit_min

Have a look at the following:
http://forum.qnap.com/viewtopic.php?t=10268
http://forum.qnap.com/viewtopic.php?t=10268
http://www.cyberciti.biz/tips/linux-...ild-speed.html
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
reliable centos nas server with raid or nas boxes which is better ? Gil@LQ Linux - Server 9 09-10-2015 05:13 AM
NAS server with FreeNas... waiting to build one. Gil@LQ Linux - Software 3 08-04-2012 12:30 PM
Looking to build a File Server (a.k.a. NAS) for use with my home network able72 Linux - Newbie 2 05-05-2012 03:59 PM
[SOLVED] Help with mdadm to build a RAID 1 array on an ARM NAS mauromol Linux - Embedded & Single-board computer 12 05-01-2012 04:21 PM
Want to build a linux based nas/media streaming server bldgengineer Linux - Server 5 07-27-2011 11:17 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Server

All times are GMT -5. The time now is 08:16 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration