LinuxQuestions.org
Review your favorite Linux distribution.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Server
User Name
Password
Linux - Server This forum is for the discussion of Linux Software used in a server related context.

Notices

Reply
 
Search this Thread
Old 11-14-2012, 11:45 AM   #1
Nick_C
Member
 
Registered: Jan 2012
Location: London, England, UK
Distribution: CentOS, Mageia, Debian, Fedora, openSUSE, Ubuntu
Posts: 230

Rep: Reputation: Disabled
Distributed RAID over iSCSI


The idea is to setup a distributed RAID array over iSCSI targets.

For example what about a couple of Linux servers each with say four drives, all published as iSCSI targets. Then one Server accessing each of these via iSCSI and connecting those targets together to create a software RAID array.

One of my concerns is the rebuild time if one of those servers is offline for a short amount of time. Would it then be necessary to rebuild the entire array from scratch or is software RAID clever enough to only rewrite sectors that have changed? My concern being that a small network glitch could otherwise potentially cause a long rebuild process.

Has anyone tried this will it work?

Thanks, Nick
 
Old 11-14-2012, 12:01 PM   #2
MensaWater
Guru
 
Registered: May 2005
Location: Atlanta Georgia USA
Distribution: Redhat (RHEL), CentOS, Fedora, Debian, FreeBSD, HP-UX, Solaris, SCO
Posts: 5,934
Blog Entries: 5

Rep: Reputation: 751Reputation: 751Reputation: 751Reputation: 751Reputation: 751Reputation: 751Reputation: 751
Haven't done it but wanted to mention:

What is "rebuilt" is never the entire array - it is the RAID set "member" (i.e. disk) that had the problem. Such a member would be rebuilt entirely I believe.

That is to say if you were doing RAID5 and had distributed it over 7 systems one of which suddenly had a glitch then only the member on that one system would be rebuilt. It would use information from the other 6 to rebuild that 1. RAID5 is designed to suffer the loss of one member without disruption. However if you lost 2 members your whole RAID set would become toast. This is why many like to use RAID6 or RAID10 instead. Both can suffer the loss of 2 members and RAID10 increases performance because it also mirrors all the data whereas RAID6 uses a second parity disk (RAID5 uses a single parity disk) equivalent.

I'm assuming your talking about one of those RAID levels rather than RAID1 since you said distributed. RAID1 can have more than 1 mirror but it is unusual.

Personally if it were me if I felt the network to the servers was not stable enough to deal with "glitches" I'd not try doing this distributed idea at all. However, its possible that iSCSI itself can suffer some level of "glitch" so that it isn't immediately RAID impacting. (e.g. maybe it gets the disk read/write request and retries sending the related packet on its own if not immediately satisfied). I haven't really worked with iSCSI so can't say that is the case but it makes sense to me that it would have some such safeguard.

Last edited by MensaWater; 11-15-2012 at 08:44 AM.
 
Old 11-14-2012, 12:38 PM   #3
Nick_C
Member
 
Registered: Jan 2012
Location: London, England, UK
Distribution: CentOS, Mageia, Debian, Fedora, openSUSE, Ubuntu
Posts: 230

Original Poster
Rep: Reputation: Disabled
Yes I was thinking about RAID 6 or RAID 1+0.

One stumbling block that I have just come accross is the extortionate price of 10Gb network switches at the moment; I had assumed that they would have been a bit cheaper than they are by now. Having the link between the RAID controller and the disks limited to 1Gb is going to be a bit of a bottle-neck.



Nick

Last edited by Nick_C; 11-14-2012 at 12:42 PM.
 
Old 11-14-2012, 04:07 PM   #4
calvinallred
LQ Newbie
 
Registered: Jan 2007
Posts: 4

Rep: Reputation: 0
What Software?

I am curious as to what software you run on your linux machines to publish them as iSCSI targets? Openfiler? Or something else?
 
Old 11-14-2012, 08:26 PM   #5
ccolumbu
Member
 
Registered: Oct 2009
Posts: 37

Rep: Reputation: 4
This already exists

Try looking into gluster, luster, and hadoop.

You would simply create a distributed raid array then export that file system via iscsi.
 
Old 11-15-2012, 01:42 AM   #6
antolepore
LQ Newbie
 
Registered: Jun 2012
Posts: 9

Rep: Reputation: Disabled
take a look also here

http://www.drbd.org/

DRBDŽ refers to block devices designed as a building block to form high availability (HA) clusters. This is done by mirroring a whole block device via an assigned network. DRBD can be understood as network based raid-1.
 
Old 11-15-2012, 03:06 AM   #7
tristezo2k
LQ Newbie
 
Registered: Oct 2009
Distribution: debian ubuntu solaris FreeBSD
Posts: 26

Rep: Reputation: 4
Quote:
Originally Posted by Nick_C View Post
The idea is to setup a distributed RAID array over iSCSI targets.

For example what about a couple of Linux servers each with say four drives, all published as iSCSI targets. Then one Server accessing each of these via iSCSI and connecting those targets together to create a software RAID array.
Hello,
I have done this using virtual machines on the lab to describe RAID principles.
However I wouldn't recomend it for a prod volume.
My thoughts are as follows.
RAID5 is designed to provide fault tolerance on disks: controller, disk logic, surface.
On this setup you are actually generating a 3+ super complex controller to access disks, much more error prone than scsi, sata or pata. Moreover, most of the storage topologies I know try to avoid that, putting disks on a single reliable target to be used by many initiators.
If physical disks attachment hardware is an issue, I would suggest some of these cheap raid towers: http://www.addonics.com/category/raid_tower.php
You can always attach them via sata to a linux box and export them over iSCSI using any software, but the error prone moving parts fragile disks will be safe with raid on the cage.

Regards,
Sebastian
 
Old 11-15-2012, 08:28 AM   #8
Nick_C
Member
 
Registered: Jan 2012
Location: London, England, UK
Distribution: CentOS, Mageia, Debian, Fedora, openSUSE, Ubuntu
Posts: 230

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by calvinallred View Post
I am curious as to what software you run on your linux machines to publish them as iSCSI targets? Openfiler? Or something else?
Most versions of Linux support LVM and iSCSI targets now, CentOS, Mageia, openSUSE etc.

Nick
 
Old 11-17-2012, 08:31 AM   #9
antolepore
LQ Newbie
 
Registered: Jun 2012
Posts: 9

Rep: Reputation: Disabled
Quote:
Originally Posted by tristezo2k View Post
Hello,
I have done this using virtual machines on the lab to describe RAID principles.
However I wouldn't recomend it for a prod volume.
My thoughts are as follows.
RAID5 is designed to provide fault tolerance on disks: controller, disk logic, surface.
On this setup you are actually generating a 3+ super complex controller to access disks, much more error prone than scsi, sata or pata. Moreover, most of the storage topologies I know try to avoid that, putting disks on a single reliable target to be used by many initiators.
If physical disks attachment hardware is an issue, I would suggest some of these cheap raid towers: http://www.addonics.com/category/raid_tower.php
You can always attach them via sata to a linux box and export them over iSCSI using any software, but the error prone moving parts fragile disks will be safe with raid on the cage.

Regards,
Sebastian
Certainly better an EMC storage in the enterprise, for the SOHO environment are fine the NAS recommended by you or bettere a QNAP NAS
 
  


Reply

Tags
iscsi, raid


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
iSCSI vs. distributed filesystems badkuk Linux - Enterprise 2 11-04-2010 03:35 AM
Distributed software raid over network dru0pa Linux - Server 4 07-28-2009 06:43 PM
LXer: Tutorial: Linux RAID Smackdown: Crush RAID 5 with RAID 10 LXer Syndicated Linux News 0 08-14-2008 11:20 PM
problem in ISCSi and RAID not working properly vaibhavp Linux - Software 1 06-19-2008 10:40 AM
changes on other iscsi nodes attached to an ext3 iscsi target not being seen sldahlin Linux - Server 1 06-07-2008 02:02 AM


All times are GMT -5. The time now is 12:14 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration