Why not a 3-way or even 4-way RAID level 1 (mirror)?
 

Why could there not be a 3-way or even 4-way RAID level 1 (mirror)? It seems every hardware (and at least the software I tested a few years ago) RAID controller only supports a 2-way mirror.

I recently tried to configure a 3ware 9650SE RAID controller. I selected all 3 drives. Then RAID 1 was not presented as an option. Only RAID 0 (striping, no redundancy) and RAID 5 (one level of redundancy, low performance). Is there some engineer who thinks "triple redundancy is a waste, so I'm not going to let them do that"? Or is it a manager?

Mirror RAID should be simple, even when more than 2 drives are used. The data is simply written in parallel to all the drives in the mirror set, and read from one of the drives (with load balancing over parallel and/or read-ahead operations to improve performance, though some of this is in question, too).

if you think that 2 drives failing is feasible, then something is surely very wrong with the hardware you're using. Raid 1 specifies 2 drives, so you can't use more than that and it still be raid 1. What you can do though is have spare devices if you have that little confidence in your drives. So if one fails, one of a pool of other drives can be used to recreate the mirror.

It's more a case of I might not be able to get to the machine that has the failure for many days. The hot spares sounds like a viable option.

Still, the logic required to implement mirroring would be trivial for N-way. It makes no sense not to have it as an option.

You do have the option of using mirroring within LVM, that does allow multiple mirrors to be created, not just one. It doesn't say in the guide here if it's limited to 2 mirrors, but may well not be practically limited at all... http://www.centos.org/docs/5/html/Cl...LV_create.html

As for RAID not supporting it, RAID needs to fit into standards, you would want to ratify a standard for that kind of RAID if it were to be implemented, and as the world has kept turning this long, there seems little chance of that ever happening.

Why is RAID 5 low performance?

I know about it more in theory than in practice, so maybe I'm missing something. Are you worried about the disk performance or the extra computation?

The extra computation is just for write isn't it? For read you have a clean copy of each sector. Compute power is pretty inexpensive (at the level of extra computations needed for RAID 5). Even for software RAID, I don't think it is inefficient.

The disk performance for read is similar to RAID 0 (and like RAID 0 normally better than non read). Large transfers distribute over 3 drives getting the same transfer rate benefits as 3 way RAID 0, and less than RAID 0 (but still some) benefits relative to head movement time.

For large writes, you would write one sector to each drive for each two sectors of actual data. That sounds twice as efficient as RAID 1.

Scattered small writes may be quite a bit uglier. But typical filesystem workloads don't include a large fraction of scattered small writes.

Yes.

Not exactly.

Obscure features that few customers want are extra opportunities to put bugs in the system. Focusing on what most customers want is usually better engineering.

Fitting into standards does not mean denying what could be done. A 3-way RAID 1 does provide better redundancy. There's no reason for a standard to say "3-way is not allowed". It can simply say "2-way is required", and let the manufacturer decide on any more than that. And maybe it was their decision and they made it. But it is silly to prohibit it.

As it stands now, RAID standardization doesn't mean much, anyway. You can't move drives from one brand of RAID controller to another. And the 3ware controllers don't even have true JBOD (oh, wait, there is no such standard ... yeah, that's helpful). I wonder if someone's zeal to be strictly compliant with RAID forced the removal of true JBOD.

I've experienced horrible performance with hardware RAID 5. I learned about the logic needed in RAID 5 and figured out the theoretical reduction in performance. Real life performance was worse, but I never found out why. I no longer use RAID 5 unless there is an extreme need for space and no need for performance (although I have found that the performance issues are not as bad as the drive factor goes up).

I would think so. But I wonder if maybe that is what is slowing down hardware RAID controllers. I'd think they'd have some awesome hardwired CPU instructions to do the parity calculations. Maybe not.

Are you speaking theoretical, or actual experienced performance? I'd think RAID 0 would perform quite well, up to the controller's capacity. RAID 1 should for multiple reads, too. RAID 5 does do much better at reading than writing, although I still see RAID 5 performing well below what it theoretically should.

Yes. Writing 4MB to RAID 0 should write 2MB to drive A and 2MB to drive B. RAID 1 has to write all the same data to both, so no performance gain there, although reading should have a lot.

It should be scattered according to the stride size in RAID 0 (similar in RAID 5). RAID 1 doesn't have a stride. So if I read 4MB, it can get 2MB from drive A and 2MB from drive B in just about any arrangement (yet in practice I see only one drive active and little more than one drive of performance). Maybe the I/O is getting broken up too much somewhere.

ISTM that 3-way RAID 1 would not be one of those.

Well, I suppose maybe they should focus on getting real life performance up to the theoretical levels, first.

I've only done significant performance testing for RAID 0 and 1 and only for a specific brand/model of fake RAID in Windows. The RAID 0 was significantly faster than JBOD (though not by as much as it should have been) in both reading and writing. The RAID 1 was slightly slower than JBOD for reading and significantly slower for writing. I expect the algorithms were incredibly stupid. With workloads of mainly read, any sane distribution of the read workload across the two drives should do much better than JBOD. Doing worse than JBOD must have taken a level of bad design I can't quite visualize despite seeing the results.

I have always assumed open source Linux software RAID could not be written with a level of stupidity approaching the above. But in fact I neither reviewed the source code nor benchmarked the performance.

I was talking about RAID 5, but you seem to be talking about RAID 0.

Writing 4MB to RAID 5 should write 2MB to each of drives A, B and C, where that 6MB total consists of the 4MB of data plus 2MB of parity.

My point was that scattered small in RAID 5 is not similar to RAID 0. In RAID 0, a small write goes to whichever one of drive A or B it happens to land on and that is it.

In RAID 5, a small write to drive A means you must also read corresponding data from drive B before writing new parity to drive C. That may be bad enough that if you expect scattered small writes to be common, you might reject RAID 5.

First: RAID1 can be used with 'n' drives, not just 2.
Second: You can also use nested levels in RAID... like making a RAID1 out of 2 (or more) RAID1 devices...
Although I personally see no logic in this, it CAN be done ;)
Third: RAID1 has a slight 'read' advantage over RAID5, which will decrease as the number of drives increases.
On the other hand, RAID5 has a 'write' advantage over RAID1, except if there are many small writes (smaller than the RAID5's stripe size).

If you value performance over size, then a RAID1+0 is the best option for you as it has the best performance in 'read' as well as 'write' of all RAID levels.
The minimum number of drives for a RAID1+0 is 4 (make a RAID0 out of 2 RAID1 (each RAID1 has 2 drives)).

I would agree RAID1 conceptually (theoretically) would/should allow this. It's implementations that I see not doing so.

Dude, I just tried it with software RAID.
It works as advertised ;)

I've seen 3, or maybe 4 by now, implementations of software RAID. Surely by now 1 or 2 of them have gotten it right? Hardware implementations, though, seem to be the big problem. I did post this in the hardware section.

You sure did...
...although...
...you did mention software implementations as well, implying that _none_ followed RAID specs correctly.

I only answered that RAID1 specs _do_ allow for more that 2 drives, and that software RAID1 on linux works as advertised, hoping it would help you in your particular case.
If you already knew software RAID1 works with more that 2 drives, then your first post is misleading :(

If hardware RAID1 is not implemented correctly in your case, you have 3 choices:
1) talk with the manufacturer of your RAID controller
2) switch manufacturer (or model) of your RAID controller
3) use software RAID, as with RAID1 there is no benefit of using hardware vs software RAID

My advice: go with number 3

Yeah, I guess I left that impression. But I have only dabbled in software RAID. Until recently, hardware (e.g. IDE) didn't let software parallel I/O very well. Now days, I've seen that do much better, to the point I can see only minimal reduction in I/O to one device when hitting another device. So maybe now software RAID is more viable. But I'm also a "keep it simple" oriented person, so doing LVM just to do RAID is not my style.

I've never seen anything in RAID specs that disallowed more than 2x redundancy in mirroring. But it's not the specs I'm concerned about; it's the implementations in hardware (so technically, not really appropriate for LQ ... just wanting to get a Linux administrator's perspective on this).

1. That's always hard to do.
2. Who has N-way mirroring and all the other stuff?
3. I have in the past seen hardware as the advantage.

I still have concerns with software RAID. One is that I/O drivers are still structured in a way that involves duplicate buffers. Has that been solved? Also, available controllers that are just plain controllers, and have high performance, seems to be limited.

Who mentioned LVM? :)
You don't need it for software RAID.
Never heard of it. Got some links?
Care to clarify?

I use software RAID on my servers because it is much more flexible.
From setting-up reporting drive failures via email so I can act faster to complex nested RAID levels... I'm in total control.
I can even make RAID devices out of partitions or block devices located on other servers, if I so desire.

I really don't see the advantage of hardware RAID over software RAID, unless I have a insanely high-end controller.
Even then, I would depend on what ever options the manufacturer 'thinks' should be available....

acid_kewpie did in post 4. Yeah, he didn't say I had to use LVM. But maybe he was saying that was the only way to do 3+-way mirroring. I do remember long ago using the MD RAID and trying to set that up and it didn't have a way to specify 3 drives for a mirror. Maybe it does, now.

I know there are other ways. I've never tried the LVM way. I don't want to, either. I like to keep life simple. A simple means to create new block devices (partitionable) from multiple other data sources is all that should be needed. Ideally, they should have integrated this with loopback devices in a more generic way, such that you have the kernel create a block device out of whatever sources you reference (other block devices including individual partitions, files, and even network block device sources).


Hmm. No. No. I only mentioned LVM to head off more suggestions to use LVM.

I have avoided it due to performance issues. Those issues were not the software implementation, but rather, were hardware issues due to the RAID being done in software.

If your controller slows down, or the drivers for it slow down, due to multiple device I/O, the software RAID can be a loser. Older controllers would readily do that, especially the cheap chip controllers used on most motherboards. Even most high-end server boards had these with the idea that if this was for a machine that needed RAID I/O performance, you'd add that. Before SATA, few, if any, motherboards even had SCSI on board ... you added that in one or more controller cards. And even then, SCSI would hurt performance unless you split things across multiple SCSI channels. Most motherboards had IDE on the board, and performance usually was bad for anything more than one device (even if there were 2 IDE channels). Some of the IDE issues were driver issues, too.

SATA came along, and as it matured, these issues have become mostly moot. Now, everyone is doing one device per SATA channel (you can do more, but it's harder to implement ... and I'm glad it has not caught on). Many, probably most now, SATA controllers can do all I/O channels pretty much independently.

Still, software RAID 1 does consume some extra RAM buffer space.

Software RAID is more practical now than it was before. I really should get back into it. But I'm also often finding that I need more than the 6 SATA channels the motherboard has. Anyone know of a high performance SATA/PCIe controller that does NOT do RAID (e.g. is cheaper than a RAID card)?

Please read my posts more carefully :)
I told you I never heard the above quoted statement you made about duplicated buffers in linux software RAID drivers, and asked for links about it, not LVM.
I know what LVM is ;)

I also did not understand the quoted statement, so asked for you to clarify.

Also, once again, I never heard of this...

You can do all of this with linux software RAID.

Something is wrong with the threading in this thread. This is not what I read. Now the posts seem to be out of order. The quoting is even inconsistent. So it seems what displayed for me and what displayed for you are not the same. But I have not been keeping snapshots of it, so I cannot go back. I'm outa here, now.

@Slax-Dude: I have edited post #15. Please, check if it appears as the original one, now. Thanks.

It does :)
Thank you so much for fixing it!