Bad RAID5 performance - horrible when reading over the network
 

Hey everybody.
I've been running a RAID5 setup on my home server for almost a year now, and it's been working fine, albeit not that fast. The numbers I get when running local benchmarks seem decent (probably a bit lower than they should be, though), but networked performance is truly horrible at times!

I've narrowed the problem down to my RAID array - if I copy from a non-RAIDed partition, the transfer rate is OK (7-8MB/sec over a 100Mbps network), but from RAID I get an average of 2-4MB/sec, it keeps jumping between 0 and 7MB/sec.
The performance is not due to the network (I've run iperf), and as stated above, performance is better when not moving from the array.

I've used three protocols when transferring over the network - Samba, AFP (Apple File Protocol) and FTP, they all share the same problem so I doubt they're at fault. OK, down to the dirty details. The computer:

Athlon XP 1700+ on a VIA KT133A motherboard (previously a Duron 700 on a KT133 MB, same problem there)
512MB SDRAM
2x Western Digital WD2000JB 200GB drives (7200rpm, 8MB cache, on the motherboard controller)
1x Seagate Barracuda 7200.7 250GB SATA drive (7200rpm, 8MB cache, on a separate PCI controller card)

mdadm -D:
50GB from the 250GB drive is used for / and such.

Misc crap:
Some numbers, and such (taken from anoter computer, moving stuff from the server):
Copying from the array (1.5-8% CPU usage)
http://static.pici.se/pictures/sLrKXVBCH.png

Coping from outside the array (~15% CPU usage)
http://static.pici.se/pictures/uZGbClUbs.png

As you can see, both read performance (48.6MB/sec RAID, vs 55.5MB non-RAID) and write performance (26.1MB RAID, vs 37.7MB non-RAID) is better outside the RAID array.
However, the main problem is the slow networked performances (see images above).

Any help is very appreciated!
BTW, I'd rather not recreate the array or do anything else destructive, I don't have enough space to back everything up at the moment.

What I/O scheduler are you using?

cat /sys/block/hda/queue/scheduler
cat /sys/block/sda/queue/scheduler
cat /sys/block/hdc/queue/scheduler

If you are not using deadline, what happens when you switch it?

echo "deadline" > /sys/block/hda/queue/scheduler
echo "deadline" > /sys/block/sda/queue/scheduler
echo "deadline" > /sys/block/hdc/queue/scheduler

I take it the change is immediate? (cat:ing again shows that it changed)
Using the same parameters to dd as above, I get 25.1MB/sec write and 46.1MB/sec read, so slightly worse. FYI I was using the anticipatory scheduler before.

Thanks for the reply :)

Edit: Using CFQ, I get 24.8MB/s write and 50.5MB/sec read.

I have severe reservations about software RAID - if you want performance, go hardware RAID.
Regardless, there has to be conflicts between what the IO schedulers are trying to do by sorting (and "optimizing" - FSVO optimizing) the IO before handing off to the VFS driver, and what the RAID is actually doing by splitting up the IO again.

I'd be inclined to try NOOP as the IO scheduler - might compromise your non-RAID IO performance, but that is a cost you'd have to prioritize.
Might be worth trying as a test.

NOOP gave me better read performance (55.8MB/sec) but the same write (26.1MB/sec).

Still, this is not the main problem. 55MB/sec is more than enough for my purposes, the real problem is that it can't even keep up with a 100Mbps networking card when reading from the array!

What was the performance over the network when using the deadline scheduler (not local). That's where you are having an issue, right?

Using deadline:
5.8MB/sec from the array (three big files)
7.32MB/sec from /dev/sda2 (on many smaller files, MP3s)
The difference doesn't look that big, perhaps, but the graphs are way different (see my first post); when moving from the single disk there appears to be a steady stream of data, as opposed to the array that locks up and stops transmitting every now and then. I'm looking to get a network upgrade, in which case 6MB/sec would be way too low.

BTW, both were measured using size/time (800MB and 1GB) so they should be accurate, average numbers.

I smell a hardware rat with this one ... your RAID and non-RAID numbers look ok given that your system has additional workload doing RAID5 checksumming and stuff - you can expect some loss in performance. What's concerning me is not your average throughput from the RAID over the network, but the lack of consistency. You say it's jumping from 0MB/s to about 7MB/s. You've already said that 7MB/s is the sustained transfer rate from the non-RAID volume, and that looks awfully similar to the maximum rate you are getting from the RAID (and is a reasonable speed for fast ethernet).

I'm wondering if there's an intermittent problem with one of the members of the raidset, or possibly some kind of bus conflict going on between the drive controllers and the NIC - either IRQ sharing or some DMA problem. It could be worth moving your cards around in their slots and see if the problem goes away, or the symptoms change.

All this is of course assuming that your NIC, VIA ATA controller and Silicon Image SATA controller aren't all on-board!

I tried moving the cards around a bit (the LAN-side NIC and the SI SATA controller), no effect unfortunately.
I also tried changing a bunch of PCI/IDE related options (PCI Dynamic Bursting, PCI Delay Transaction, PCI #2 Access #1 Retry and IDE Prefetch Mode), all from disabled to enabled. No effects changing PnP OS from No to Yes either, I was hoping that might change some IRQs around.

Like a girlish band once sang... What's going on!?!
This makes ZERO sense to me!
OK. I tried using the other NICs for my LAN, that is, move the LAN cable to eth0 (from eth1) and set it up. OK, so I did that. Hmm. SSH seems slow... So I tried a file transfer. I got 587kbps!! Yes, that's right, five hundred eighty-seven kilobits per second. What the heck, that NIC downloads at ~800kB/sec (that's kilobytes) from the internet daily!
OK, so I tried the OTHER NIC (eth2), started a file transfer... WHOA, 18MB/sec! Over a 100Mbps NIC, this can't be! And it wasn't, either. I did the usual time-a-file and the average I got was... 3.2MB/sec. Both nload on my Linux box AND my laptop were both reporting the ultra-high speeds!

I'm going to keep looking for a cheap replacement box, while doubting my sanity...
Help is still appreciated though, I'm not giving up that easy!

will confirm that nothing's sharing IRQs and should completely eliminate anything in that area.

I've never used the 3c509 with Linux, but it's a decent branded card and I wouldn't expect it to cause problems. The Silicon Image card is my favourite SATA controller. All my own servers and a good percentage of my customers' Linux servers use them, and I've not had any issues with it.

What I've never done is mix ATA and SATA drives in a single RAID set. Now config issues are eliminated, I would suspect there's something going on there, although diagnosing it with live drives is not going to be easy. It appears that something is getting deadlocked somewhere, but then releasing which could indicate kind of buffering issue. Are you running SMART monitoring? It works with ATA drives, but not SATA drives - could be something there. Reading through this thread, I'm somewhat clutching at straws.

That is very weird. Could be like some kind of PCI bus mastering thing.

What I would do, is make sure the SATA controller and the primary NIC that you need the performance through are in PCI bus mastering slots - if there are 5 slots on the board, there may only be a couple that support this.

You seem to be heading down the right path. Are all the NICs the same 3c509? If so, you could try a different brand of NIC, and see if this has any effect.

Hmm. eth0 and eth1 are sharing IRQ 5 as it stands right now.
Two of the three NICs are 3c905b's. Currently, the non-3c905b is the used for the LAN, but the weird results (one uber-slow, one seemingly uber-fast) were both from the different 905 NICs! Very odd.

In any case, I mentioned looking for a computer to a friend who offered me an irresistable deal, so I'll simply get that and hope that the problems are gone (the HDDs will be the only thing the computers have in common). Don't get me wrong though, this thread isn't the only reason why I'm upgrading, as I said I'm not giving up this easily. :)

Also... Yes, I am in fact running smartmontools (using the -d ata option for the SATA drive, works great). I'm pretty certain the performance was this bad waay before I even installed that, but not 100%.

Edit: According to MSI "Five 32-bit Master PCI bus slots."
I'm guessing that means bus master.

Weirdness - it does look like there is some kind of deadlock condition happening between the NICs and one of the ATA controllers (I think we can discount the SATA controller as you've tested that separately with no problems). Could be something hogging the PCI bus, could be a buffering issue.

Yes - it's annoying when you can't quite figure what's happening. Never give up without a fight, but there comes a time where for the sake of your sanity, you just have to accept the weirdness, back up the data, flatten the box and start over. The new system will allow you to figure it out without the risk of losing your data.

You could try temporarily disabling it on startup, just to eliminate it as a suspect.

Some of those old VIA KT133 boards were great! Really feature packed for the money! :cool: