Actual bandwidth vs rated

bradvan · 03-21-2013, 10:26 AM

I have an HP c7000 blade system with some BL460 G5 and G7 blades and a couple of VC FLEX-10 switches. The blades are running RHEL 6 and the current network is running at 5000Mb/s full duplex. I created a 10GB file and scp'd it from one blade to another. I got an aggregate of about 50MB/s throughput. 5000Mb/s should correspond to 625MB/s. I know I'm not going to get the whole bandwidth, but shouldn't be getting a little more than about 12% of the bandwidth?

suicidaleggroll · 03-22-2013, 02:14 PM

As far as I can tell, those blades only have 1GbE ports...are you running a separate 10GbE network card in them? What makes you think your network is running at 5Gb?

Also, the SCP encryption has a lot of overhead. Check top while running it, if the processors in the machines aren't up to the task you'll be limited by their encryption/decryption speed. You can try changing the SSH cypher to blowfish to reduce processor overhead, assuming that's your bottleneck...otherwise you'll need to change protocols.

baldy3105 · 03-22-2013, 06:59 PM

Test it first with iperf. Memory to memory over the network, removing all the IO subsystems from the equation and issues with encryption etc. Bear in mind that if you are running port channeled gig links, and I assume you are if you have a 5G connection to the network, you will still be limited to 1Gbps for a single session, you need multiple sessions to make use of multiple links.

bradvan · 03-23-2013, 10:09 AM

Hi guys, I'll give iperf a shot. It is a 10Gb on board NIC. The flex-10 splits that into two 5Gb connections. So, the single NIC is on a 5Gb connection and it is not bonded. I was thinking of bonding the two.

I'll run top at the some time. I can also try an nfs connection and remove the encryption overhead.

Thanks for the suggestions!

jefro · 03-25-2013, 03:30 PM

My guess would be that you could never get full bandwidth on that nic. Either backplane/cpu or hard drive would be slower than nic. If you had some huge file in ram or ramdrive then maybe it might get close to or maybe 70% of simple transfer speeds.

suicidaleggroll · 03-25-2013, 06:56 PM

Quote:

Originally Posted by jefro

If you had some huge file in ram or ramdrive then maybe it might get close to or maybe 70% of simple transfer speeds.

Good point, regular platter drive sequential read is only around 100 MB/s (give or take), so in order to exceed that with any kind of file transfer you're going to need to be pulling off of a ramdrive, SSD, or a RAID array.