Very slow (Gigabit crossover - pc to pc) network connection!
 

Hello All..

I've installed kubuntu before 3 days on one of my PC, for which I bought one more NIC. I have a small home network of 3 PCs. Have a linksys wireless DSL modem/router. The router and the third PC are not involved in the problem. Wanted to have very fast connection between the 2 PCs, so I decided to use second Gigabit NICs to directly connect them.


Some more info about my setup:

The first PC Celeron 1.3 (PC A) running kubuntu 5.10 has 2 NICs:
1. D-Link System Inc RTL8139 connected to the router for internet access.
2. "Noname" RTL-8169 (Gigabit ethernet) for peer to peer connection to my second PC (PC B).

The second PC (PC B) Athlon64 3200 on a motherboard having the nvidia4 chipset running Win2kPro (OS, Win2k NOT involved to the problem), as I' ve been using an Ubuntu 5.10 live CD on PC B to verify the problem and run my tests (to make sure it is not win related). PC B has 3 NICs:
1. A "Noname" RTL-8139/8139C/8139C+ connected to the router for inet access.
2. Onboard Nvidia Gigabit NIC for peer to peer connection to PC A.
3. PCI Netgear wireless card (not in use)

I'm using a new crossover CAT5e cable (1 meter long) to connect:
PC A - RTL8169 NIC <-> PC B - Onboard Nvidia NIC.
and this connection is very slow,
Also tried some simple CAT5 cable (Gigabit NICs can work with or without crossover cables).

Using an Ubuntu live CD on PC B, I've done many tests transfering a large file (created with dd if=/dev/zero ...) between PC A and B. For the tests I've been using scp. To monitor NICs speed I've been using iftop on PC A and the output of the scp command (scp always running on PC B).

Here are some results scp-ing my file:
PC B -> PC A: 8.0 - 8.6 MB/sec (about 68 Mbps).
PC A -> PC B: 9.1 - 9.5 MB/sec (about 77 Mbps).

HD speed in not a bottleneck, PC B has a new SATA drive. Testing PC A's hd speed suggested that 30MB/sec seems possible.

I've changed the MTU on both gigabit NICs to 4500 and tried again:
PC B -> PC A: 10,6 MB/sec (about 85.8Mbps)
PC A -> PC B: 9.4 - 9.7 MB/sec

Running the same tests, but with the fast ethernet NICs (RTL8139 cards connected through the router) gave:
PC B -> PC A: about 8.5 MB/sec
PC A -> PC B: 7.6 - 8.1 MB/sec (about 66Mbps).

I've also run the same tests connecting Gigabit NICs to Fast NICs, which gave about 8.5MB/sec.

While doing the tests I've brought all other (not in use) network interfaces down on both PCs.

I've also done the test with and without using a file /etc/modprob.d/r8169 on PC A (Kubuntu) containing:
alias eth0 r8169
options eth0 use_dac=0
but same results.

On the LiveCD Ubuntu I've been using linux acpi=off to boot.


All started from something worse... While using samba to transfer files from Kubutu (PC A) to Win2k (PC B) speed is about 1,5MB/sec! And I need to transfer about 150Gb across these PCs these days! Shocked

Here is the output of some useful commands:
Thank you in advance for your help!
gecon

What do do the routing tables look like on both computers?

You might try using iperf (google it) rather than scp. There's probably some encryption overhead associated with scp, but admittedly, your performance should be better than 10Mb/s over a gigabit link.

I've got an 8169 PCI NIC, too. Later this evening I'll run some iperf tests to see what I get so we can compare.

Jay

I don't believe route tables has anything to do with my problem, but here they are...

Follows PC A routing:
PC B routing is the same... (gateway is the router, eth0 is the gigabit card on both PCs)

jcliburn: I'm aware of the ssh encryption overhead, but I agree with you, 10Mb/sec on a Gbit NIC?! Thank you, I will run some tests with iperf, for comparisson.


gecon

The issue I'm wondering about is that your performance for a 1Gb card is terrible but is right on for 100Mb card. You've got two networks (a 1Gb ans a 100Mb) and both machines can see each other directly across both networks. This may be causing some confusing in the routing and the traffic may very well be going out eth1 even though you want to go out eth0.

centauricw: This is why I brought down the second (not in use) interface (eth1) while doing the tests...

Using iperf, I'm getting about 437 Mb/s (average) between a VIA Velocity 6122 gigabit NIC and a Netgear GA311 (RTL-8169) gigabit NIC. There are 2 gigabit switches between the hosts. Host 192.168.1.3 (osprey) has the VIA NIC, and host 192.168.1.10 (gadwall) has the Realtek NIC.

Here's the client side.
Here's the server side.

For what it's worth, I forced my Realtek 8169 NIC to 100 Mb/s and got numbers very close to yours.
This seems to point to the possibility that your NICs are autonegotiating down to 100 Mb/s. Try using ethtool to disable autonegotiation on one of the links. Chances are, your transfer won't work at all. This would imply a bad NIC or a bad cable, most likely the latter. Stupid question perhaps, but are you sure you're using a cat5e crossover? Do you have a way to validate the cable between two other systems that are known to work at gigabit speeds?

Jay

jcliburn: Thank you!!
Ok.. Latest news. NICs are working fine, problem remains..

Testing with iperf I've got 321 Mbits/sec on average which is ok. On some test I've got even >400Mbits/sec, but 312Mbits/sec is valid (for me) for a Gigabit connection.

So, what can this be, really?
Retested scp and samba after iperf tests and got the same old slow speed. (scp ~ 10MB/sec, Samba ~1,5MB/sec).

Tested hd performance again:
Don't know why hdb is so slow (primary slave with ntfs filesystem, but isn't fs type irrelevant when using dd?) compared to hda (hdb is quite new h/w compared to hda!), but both hd's are fast enough to provide better speed to Gigabit NICs transfers (scp trasfer test with hda):
hda ~ 60MB/sec, hdb ~ 22MB/sec

Any ideas?? Why are transfers so slow since iperf gives normal Gbit speeds?

gecon

1. I' ve run some tests using ftp and got about 160 Mbps speed on large file upload from PC B to PC A.
2. I' ve run ifperf -F <file> to have a file for input. As before, iperf gave good results (>320Mbps).

The next one thanks to your suggestions jcliburn:
3. There seems to be some strange behavior when I try to change settings (speed, autonegotiation) on rtl8169 on PC A (kubuntu), which is not seen when changing settings of the fast (rtl8139) card. Here it is (eth0 is the rtl8169 card):
I enter:
and after that I run the "ethtool eth0" command.
The auto-negotiation of eth0 is reported: on. Always. Never shows 'off'.
I mean the following lines of the output:
are always there, regardless of speed setting, always 'Yes', 'on'.

On my fast rtl8139 card after setting "autoneg off" the "ethtool eth1" command shows 'No', 'off' for the above fields.

Moreover the following is strange (!):
and after the above I run ifperf tests that gave a lot of variation. Results on many runs:
122 Mbps, 157Mbps, 395Mbps, 218Mbps, 203 Mpbs, 394Mbps, 132Mbps, 322 Mpbs
and even no connectivity, I've had to /etc/init.d/networking restart to find network.
:eek:

If I enter:
the iperf results are typical(?) for fast ethernet (82, 79, 81, 73, 91 Mpbs)


If I enter:
the iperf results are quite good (but the problem with apps other than iperf remains). iperf gives: 333, 295,358, 357, 292, 379, 346

And finaly if I enter:
There is no link established. :(
jcliburn: I can not validate the cable as I don't have access on other Gbit devices. I bought the cable new and it says cat5e on it.
Do you think it is the cable, the NIC or maybe something else...
:confused: :(


gecon

I' ve tested another cable which is CAT5 (not CAT5e) and it has the same behavior as the CAT5e I have which is supposed to be for Gigabit??
Ok, the questions now are:
- If the cable and the cards are ok, should a connection be established (link) always when using:
?
- The fact that autoneg off is not reported as set from ethtool, suggests a software and/or hardware (NIC) problem?
- How come the ifperf report such good speeds (over 100Mbps difference) compared to every other traffic?

gecon

Just a quick tip about gigabit ethernet. In gigabit ethernet the transfer uses all 4 pair (not like 10/100 1,2,3 and 6) so if you make a crossover cable for gig, i assume you changed pins 1,2,3, and 6 causing the card to stay at 100. There might be a way to roll the cable but in all my gigabit network a device is needed to get full through-put. cat5 cable is good for gig also.

dclark: The cable is CAT5E bought from a store (I have not made it).
Gigabit ethernet NICs can operate and establish a link with plain cable (not crossed over) even in direct connection (peer 2 peer) between the NICs.
I only have 1 CAT5e Cable here (the one which states "crossover").
I also have tried 2 different CAT5 cables (no crossover) which gave the same behavior as the CAT5e crossover cable.

gecon

My results are similar to yours whenever I dork with the 8169 settings. As a matter of fact, I need to walk upstairs and reset autoneg back to "on" after I lost the link by setting speed=1000 and autoneg=off.

I think you've got a good NIC and a good cable. scp performance simply sucks, I guess, owing to the encryption. I see the same thing transferring a file full of floating point numbers I ginned up.

jcliburn: I see..
But I' m wondering, if so, the 1,5MB/sec of Samba is usual/"high" rate? :eek:

Thank you jcliburn!
gecon