The idea is to setup a distributed RAID array over iSCSI targets.

For example what about a couple of Linux servers each with say four drives, all published as iSCSI targets. Then one Server accessing each of these via iSCSI and connecting those targets together to create a software RAID array.

One of my concerns is the rebuild time if one of those servers is offline for a short amount of time. Would it then be necessary to rebuild the entire array from scratch or is software RAID clever enough to only rewrite sectors that have changed? My concern being that a small network glitch could otherwise potentially cause a long rebuild process.

Has anyone tried this will it work?

Thanks, Nick

Haven't done it but wanted to mention:

What is "rebuilt" is never the entire array - it is the RAID set "member" (i.e. disk) that had the problem. Such a member would be rebuilt entirely I believe.

That is to say if you were doing RAID5 and had distributed it over 7 systems one of which suddenly had a glitch then only the member on that one system would be rebuilt. It would use information from the other 6 to rebuild that 1. RAID5 is designed to suffer the loss of one member without disruption. However if you lost 2 members your whole RAID set would become toast. This is why many like to use RAID6 or RAID10 instead. Both can suffer the loss of 2 members and RAID10 increases performance because it also mirrors all the data whereas RAID6 uses a second parity disk (RAID5 uses a single parity disk) equivalent.

I'm assuming your talking about one of those RAID levels rather than RAID1 since you said distributed. RAID1 can have more than 1 mirror but it is unusual.

Personally if it were me if I felt the network to the servers was not stable enough to deal with "glitches" I'd not try doing this distributed idea at all. However, its possible that iSCSI itself can suffer some level of "glitch" so that it isn't immediately RAID impacting. (e.g. maybe it gets the disk read/write request and retries sending the related packet on its own if not immediately satisfied). I haven't really worked with iSCSI so can't say that is the case but it makes sense to me that it would have some such safeguard.

Yes I was thinking about RAID 6 or RAID 1+0.

One stumbling block that I have just come accross is the extortionate price of 10Gb network switches at the moment; I had assumed that they would have been a bit cheaper than they are by now. Having the link between the RAID controller and the disks limited to 1Gb is going to be a bit of a bottle-neck.



Nick

I am curious as to what software you run on your linux machines to publish them as iSCSI targets? Openfiler? Or something else?

Try looking into gluster, luster, and hadoop.

You would simply create a distributed raid array then export that file system via iscsi.

take a look also here

http://www.drbd.org/

DRBD® refers to block devices designed as a building block to form high availability (HA) clusters. This is done by mirroring a whole block device via an assigned network. DRBD can be understood as network based raid-1.

Hello,
I have done this using virtual machines on the lab to describe RAID principles.
However I wouldn't recomend it for a prod volume.
My thoughts are as follows.
RAID5 is designed to provide fault tolerance on disks: controller, disk logic, surface.
On this setup you are actually generating a 3+ super complex controller to access disks, much more error prone than scsi, sata or pata. Moreover, most of the storage topologies I know try to avoid that, putting disks on a single reliable target to be used by many initiators.
If physical disks attachment hardware is an issue, I would suggest some of these cheap raid towers: http://www.addonics.com/category/raid_tower.php
You can always attach them via sata to a linux box and export them over iSCSI using any software, but the error prone moving parts fragile disks will be safe with raid on the cage.

Regards,
Sebastian

Most versions of Linux support LVM and iSCSI targets now, CentOS, Mageia, openSUSE etc.

Nick

Certainly better an EMC storage in the enterprise, for the SOHO environment are fine the NAS recommended by you or bettere a QNAP NAS