Slow gigabit performance with software RAID
 

alright i've searched for a good deal and haven't been able to find an answer to my problem so hopefully you guys can help. i recently setup a software raid 5 array on my slackbox. it works great copying from drive to drive locally in the computer. the problem comes when i try to copy files across the network. when i try to do that, the speeds are much slower than my gigabit card gave me before i added the raid to the system. at the same time i don't see where the software raid could plug up the system enough to slow transfer on my gigabit adapter to 3MB/s where from local drive to local drive its 30MB/s. also before establishing the raid array i could get sustained transfer rates around and above 30MB/s so i know the adapter works well.
on a side note, upon upgrading my kernel and getting this thing running i began to have some errors when i first tried to copy things over the network. my system would hard lock then on reboot i would get an error like

APIC error on CPU0: 00(02)
APIC error on CPU1: 00(02)

in dmesg. i then added this line

to my lilo.conf and it stopped the errors and lockups but i was unable to evaluate network performance before so i'm unsure if that could also be hurting the system.

i'm kinda stumped due to the fact that if my pci bus were being saturated, i figure it would show with the drive to drive transfers as well as the network ones, and i know the adapter is functioning properly. any ideas?-The_Nerd


Edit:
i have been trying things all day. i found earlier that my cpu usage was about 30% while resting and i thought the problem may have had smoething to do with that. after waiting about 45 minutes the cpu usage went down to normal levels and i tried to transfer files again. now i can get speeds of about 7MB/s from a gigabit adapter. While doing two transfers across the networks, both seem to top out at about 5.2MB/s (both from the same ethernet adapter). while doing one lan and one local transfer, the transfer from the lan to the array is about 5MB/s while the local drive to array is about 25MB/s. cpu usage now during LAN transfer is about 45% on both cpu's, goes up to around 90% on dual simultaneous transfers. basically i've had no luck getting the gbe adapter to transfer files around 20MB/s which is a bit slower than it was before but it seems the raid array doesnt go much faster than that locally. anyone have any ideas? would it be worth trying a 2.6.x kernel and giving that a try?

Gigabit ethernet should be on a 64-bit PCI bus or better(PCI-X). RAID should be on a hardware controller(3ware if PATA/SATA). APIC should be disabled in the motherboard BIOS. PNP should be disabled in the motherboard BIOS.

You will saturate your link with a 32-bit PCI bus and Gigabit. You will saturate it even more with CPU and PCI chipset dependent software RAID. Kernel 2.6 may help with updated software RAID. It's free, so give that a try first. You didn't state what hardware you are using, so I can't help you any more than the obvious stuff.

Thoreau-
I have never had too much of a problem putting gigabit nic's in 32bit slots, all the other computers on my network only have 32bit pci and thats where the nics are, so i figured it wouldnt be that much slower or faster than the rest of the network. i actually did move the gigabit card to the second of the two 64bit pci slots and it performed worse than in the 32bit slots, but then again this was while the array was rebuilding in the backround, i havent tried it since. so let me try to move the gigabit card to the 64bit pci slot and if that doesnt work i'll try a 2.6.x kernel and be back with the results. heres my system specs too:

MSI K7D Master-L
Dual AMP 1600+
1024MB DDR
Dlink DGE 530t <-- Gigiabit etherent card
Highpoint Rocket Raid 454 <--Raid Card (i'm only using it as a controller for the drives, the array is configued in linux)

I know a lot of the components arent the best, the raid card isnt a hardware raid card as it is and i only have two 64bit pci slots, but i'm trying to cut down on my computer spending so i can pay for college:rolleyes: hopefully i'll be able to get decent performance with what i already have-The_Nerd

Edit:
i've been playing with things a lot more today. i moved the gigabit card to a 64bit pci slot and was gettng about 13MB/s which is slightly better. perhaps more interesting though is what is happening right now. i'm copying files from a single drive to a single drive. the source drive is on the raid controller (i have 3 drives in raid 5 on the controller, then a single drive on the 4th ide connector. all drives are master on the channels) and the destination drive is on the motherboard. occasinally i get transfer rates around 20MB/s, but it hovers mainly in the 7MB/s to 10MB/s range. so mabye the bottleneck is on the board? any good way to find out for sure?

Edit2:
The gigabit card in the 64bit slot seems to have about the same speeds as it did in the 32bit slot. the 13MB/s that i listed above was reading off of the array, i still get about 7MB/s writing to the array. i am going to compile kernel 2.6.11.7 in a few minutes and give that a try as well

OK, also keep in the back of your mind that you may not have enough power. Multiple drives on dual proc's and a gig link tend to suck a good bit of juice. Ghetto taiwanese 350 watt PS's aren't going to cut it. The mathematical limit of 32bit PCI will give you about half the speed of a Gig-E link. A 64-bit PCI slot will allow you to use your entire Gig-E link. PCI-X will allow you to use 2 Gig-E links per slot.

As an example, I have a 4 disk PATA array with hardware RAID on one workstation. Athlon XP 3200+. I had drive failures every few months and couldn't figure it out, since everything I own is on backup conditioned power. I was running this on a 4 oz. 450 watt PS made in some hole in China. I slapped in a Thermaltake 560 and my drive speeds literally doubled. All of my drive speeds doubled, including an external usb 2.0 250GB drive. My drives stoppped failing. And now I'm a happy boy. Just a thought.

hmm an interesting thought. the computer now is running an antec truepower 350watt which seems to have been doing me well.. but in actuality i havent checked the stability of the power rails ever. i have a spare 420watt i can try and i believe a friend of mine has a 520 i could test it with as well... i havent compiled the 2.6.x.x kernel yet but i should have time later tonight-The_Nerd

Edit:
oh yea thats a 350watt running
6 harddrives
1 cdrom drive
2 athlon mp 1600+
nvidia tnt2 agp
2 120mm fans

i've never really seen an underpowered computer so i wouldnt know any symptoms of such a situation.. any tell tale signs? just out of curiosity.. would a random error when loading the kernel be such a symptom? i was rebooting today and was met with a "kernel failed crc check" message from lilo. i shut down, and waited a few seconds, then rebooted and it was fine, but it seemd kinda odd.

The symptoms are slow drive speeds, programs under heavy load failing or segfaulting, data corruption due to not being able to write completely, symptoms akin to overclocking where stability problems are constant.

A 350 is way to small for that setup. I didn't know you had 6 hard drives. I'm amazed it boots actually. The only reason why I think it works is that antec makes good PS's, and the 350 is probably putting out 400 watts actual. You'll want to look at either removing drives are getting a 500 or better name brand PS. Antec, thermaltake, and pc power and cooling are the better one's.

now that you mention it.. i would get segfaults during some file transfers and i just kinda wrote them off. i'll be on the look out then for a new power supply... though after this kernel compile finishes i may try my 420watt generic but it may not be much better than the antec 350 thats in there right now. i'll see what i can scrape up in terms of power supplies and i'll see if the new kernel does anything but it sounds like my power supply is the culprit. thanks again Thoreau for your help.. i'll see what else i can do and post back in case anything new arises.-The_Nerd

Edit:
oh yea and this morning it wouldnt turn on the first two times i tried it.... underpowered psu seems to fit the problem description pretty well

Edit2:
thermaltake psu

any opinions on that psu? think 480watts will be enough or should i go for a 500+ watt one

Do a pricewatch.com search on W0023 or W0023R.

You have 2 cpu's and 6 hard drives a CD-ROM and 2 huge case fans. A 450 is what you use on a standard PC. This is not a standard PC. A 350 is what you use on slimline economy PC's. I suggest 500 or above.

The thermaltake 560 is 100 bucks. It will not let you down, and will work with any configuration whether dual proc, or single or SATA or PATA. It will support at least 8 drives fully powered, because I have it in my workstation.

It is only surpassed by it's slightly ungraceful goliath sister, the W0049 at 680 Watts of beast insane power. That is too much for you. The 560 is a beautiful peace of hardware and weighs 3 lbs where most wiegh 3 ounces. It is a tank, which is what you want when minipulating power.

The choice is yours.

found the W0023R at monarch computer for about $120 with shipping which seems to be pretty good. i did find the W0023 but i figured if i was getting a good powersupply i may as well go with Active PFC and go all out. as soon as i get the money (im fixing a good deal of computers right now so i should have it soon) i'll be able to get it and probably fix my problem. i did try kernel 2.6.11.7 and it didnt do anything to improve performance but i think at this point its very safe to say that the power supply is the issue. thanks again Thoreau-The_Nerd

Rock on bud. Good luck.