terribly slow file transfer rates

explotion · 12-18-2012, 05:58 PM

Hi,

I have a very persistent:
Any attempt to transfer files between servers ends up with 500-900kbps rate.
iperf test show very unstable rates around 20-100mbps. It NOT a HD bottleneck issue.
one weird thing did happen - when running over UDP it seem to get to 100%.
I tried transfer over UDP but couldn't find a proper tool for that (ones i tried worked slow)

The servers have 10Gbps intel NIC at the source and 1Gbps at destination
they are in two different locations and run on CentOS 6.
The server reaches couple of Gb/s when there are a lot of users.
but when trying to simple sequential transfer it works slow!!

I have tried virtually everything:
rsync with various ssh encryption
scp
tar
wget
tcp /udp
tftp
http
Duplexing, jumbo frames

does anyone have an idea what could cause this and how can it be solved?

Thanks for the help

nikmit · 12-20-2012, 10:22 AM

If the connection is very volatile it is probably causing lots of retransmits, which slow down the transfer. You can check that with tcpdump. Look for out of order packets, retransmits, tcp window size reductions.
It could also be down to traffic shaping from your provider. Try a transfer over a vpn - openvpn for example. This should fool most shapers.

baldy3105 · 12-20-2012, 01:08 PM

If its 100% with UDP your problem is almost certainly TCP Window related. Two questions, 1. what latency exists between the two machines and 2. Have you made any changes to the default TCP window settings on your OS's?

If not its pretty much guaranteed that you need to adjust your TCP Windowing.

There are lots of docs you'll find if you google.

explotion · 12-21-2012, 06:43 AM

Thanks for the help.

I will read on TCP windowing.

I just tried to use FTP and it seem to work with acceptable speeds.

tonyfreeman · 12-21-2012, 10:09 PM

I had a similar issue come up twice ... but I have three recommendations for you:

First though is type '/sbin/ifconfig' and have a look at the reported errors:

Code:

tony@toshiba:~$ /sbin/ifconfig
eth0      Link encap:Ethernet  HWaddr 00:1b:38:b0:76:65  
          inet addr:192.168.7.27  Bcast:192.168.7.31  Mask:255.255.255.240
          inet6 addr: fe80::21b:38ff:feb0:7665/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:2004651 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1887183 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:2095685175 (1.9 GiB)  TX bytes:348242902 (332.1 MiB)
          Interrupt:43 Base address:0x6000

If you have errors, dropped, or overruns then maybe one of the following will help you:

1) A switch was bad. There was too much chatter on the port in which the network cable was plugged into. I replaced the switch with a new one and my system sprung to life very quickly!

2) The cable I was using was bad ... actually I made it wrong. If you made the cable yourself, clip both ends and try again.

3) I used ethtool to set the negotiation manually on one of my network cards ... this was bad ... I re-set it back to auto-negotiation and the network stated working again.

.