Very slow (Gigabit crossover - pc to pc) network connection!
Linux - NetworkingThis forum is for any issue related to networks or networking.
Routing, network cards, OSI, etc. Anything is fair game.
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
Very slow (Gigabit crossover - pc to pc) network connection!
Hello All..
I've installed kubuntu before 3 days on one of my PC, for which I bought one more NIC. I have a small home network of 3 PCs. Have a linksys wireless DSL modem/router. The router and the third PC are not involved in the problem. Wanted to have very fast connection between the 2 PCs, so I decided to use second Gigabit NICs to directly connect them.
Some more info about my setup:
The first PC Celeron 1.3 (PC A) running kubuntu 5.10 has 2 NICs:
1. D-Link System Inc RTL8139 connected to the router for internet access.
2. "Noname" RTL-8169 (Gigabit ethernet) for peer to peer connection to my second PC (PC B).
The second PC (PC B) Athlon64 3200 on a motherboard having the nvidia4 chipset running Win2kPro (OS, Win2k NOT involved to the problem), as I' ve been using an Ubuntu 5.10 live CD on PC B to verify the problem and run my tests (to make sure it is not win related). PC B has 3 NICs:
1. A "Noname" RTL-8139/8139C/8139C+ connected to the router for inet access.
2. Onboard Nvidia Gigabit NIC for peer to peer connection to PC A.
3. PCI Netgear wireless card (not in use)
I'm using a new crossover CAT5e cable (1 meter long) to connect: PC A - RTL8169 NIC <-> PC B - Onboard Nvidia NIC.
and this connection is very slow,
Also tried some simple CAT5 cable (Gigabit NICs can work with or without crossover cables).
Using an Ubuntu live CD on PC B, I've done many tests transfering a large file (created with dd if=/dev/zero ...) between PC A and B. For the tests I've been using scp. To monitor NICs speed I've been using iftop on PC A and the output of the scp command (scp always running on PC B).
Here are some results scp-ing my file:
PC B -> PC A: 8.0 - 8.6 MB/sec (about 68 Mbps).
PC A -> PC B: 9.1 - 9.5 MB/sec (about 77 Mbps).
HD speed in not a bottleneck, PC B has a new SATA drive. Testing PC A's hd speed suggested that 30MB/sec seems possible.
I've changed the MTU on both gigabit NICs to 4500 and tried again:
PC B -> PC A: 10,6 MB/sec (about 85.8Mbps)
PC A -> PC B: 9.4 - 9.7 MB/sec
Running the same tests, but with the fast ethernet NICs (RTL8139 cards connected through the router) gave:
PC B -> PC A: about 8.5 MB/sec
PC A -> PC B: 7.6 - 8.1 MB/sec (about 66Mbps).
I've also run the same tests connecting Gigabit NICs to Fast NICs, which gave about 8.5MB/sec.
While doing the tests I've brought all other (not in use) network interfaces down on both PCs.
I've also done the test with and without using a file /etc/modprob.d/r8169 on PC A (Kubuntu) containing:
alias eth0 r8169
options eth0 use_dac=0
but same results.
On the LiveCD Ubuntu I've been using linux acpi=off to boot.
All started from something worse... While using samba to transfer files from Kubutu (PC A) to Win2k (PC B) speed is about 1,5MB/sec! And I need to transfer about 150Gb across these PCs these days! Shocked
Here is the output of some useful commands:
Code:
PC A:
root@oliaros:~# ethtool eth0
Settings for eth0:
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Advertised auto-negotiation: Yes
Speed: 1000Mb/s
Duplex: Full
Port: Twisted Pair
PHYAD: 0
Transceiver: internal
Auto-negotiation: on
Link detected: yes
root@oliaros:~# lspci
0000:00:00.0 Host bridge: VIA Technologies, Inc. VT82C693A/694x [Apollo PRO133x] (rev c4)
0000:00:01.0 PCI bridge: VIA Technologies, Inc. VT82C598/694x [Apollo MVP3/Pro133x AGP]
0000:00:07.0 ISA bridge: VIA Technologies, Inc. VT82C686 [Apollo Super South] (rev 40)
0000:00:07.1 IDE interface: VIA Technologies, Inc. VT82C586A/B/VT82C686/A/B/VT823x/A/C PIPC Bus Master IDE (rev 06)
0000:00:07.2 USB Controller: VIA Technologies, Inc. VT82xxxxx UHCI USB 1.1 Controller (rev 1a)
0000:00:07.4 SMBus: VIA Technologies, Inc. VT82C686 [Apollo Super ACPI] (rev 40)
0000:00:09.0 Multimedia audio controller: Creative Labs SB Live! EMU10k1 (rev 0a)
0000:00:09.1 Input device controller: Creative Labs SB Live! MIDI/Game Port (rev 0a)
0000:00:0b.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL-8169 Gigabit Ethernet (rev 10)
0000:00:0c.0 Ethernet controller: D-Link System Inc RTL8139 Ethernet (rev 10)
0000:01:00.0 VGA compatible controller: Matrox Graphics, Inc. MGA G400 AGP (rev 03)
root@oliaros:~# uname -a
Linux oliaros 2.6.12-10-386 #1 Thu Dec 22 11:37:10 UTC 2005 i686 GNU/Linux
PC B:
root@ubuntu:~# ethtool eth0
Settings for eth0:
Supported ports: [ MII ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Advertised auto-negotiation: Yes
Speed: 1000Mb/s
Duplex: Full
Port: MII
PHYAD: 1
Transceiver: externel
Auto-negotiation: on
Supports Wake-on: g
Wake-on: d
Link detected: yes
root@ubuntu:~# lspci
0000:00:00.0 Memory controller: nVidia Corporation: Unknown device 005e (rev a3)
0000:00:01.0 ISA bridge: nVidia Corporation: Unknown device 0050 (rev a3)
0000:00:01.1 SMBus: nVidia Corporation: Unknown device 0052 (rev a2)
0000:00:02.0 USB Controller: nVidia Corporation: Unknown device 005a (rev a2)
0000:00:02.1 USB Controller: nVidia Corporation: Unknown device 005b (rev a3)
0000:00:04.0 Multimedia audio controller: nVidia Corporation: Unknown device 005
9 (rev a2)
0000:00:06.0 IDE interface: nVidia Corporation: Unknown device 0053 (rev f2)
0000:00:07.0 IDE interface: nVidia Corporation: Unknown device 0054 (rev f3)
0000:00:08.0 IDE interface: nVidia Corporation: Unknown device 0055 (rev f3)
0000:00:09.0 PCI bridge: nVidia Corporation: Unknown device 005c (rev a2)
0000:00:0a.0 Bridge: nVidia Corporation: Unknown device 0057 (rev a3)
0000:00:0b.0 PCI bridge: nVidia Corporation: Unknown device 005d (rev a3)
0000:00:0c.0 PCI bridge: nVidia Corporation: Unknown device 005d (rev a3)
0000:00:0d.0 PCI bridge: nVidia Corporation: Unknown device 005d (rev a3)
0000:00:0e.0 PCI bridge: nVidia Corporation: Unknown device 005d (rev a3)
0000:00:18.0 Host bridge: Advanced Micro Devices [AMD] K8 NorthBridge
0000:00:18.1 Host bridge: Advanced Micro Devices [AMD] K8 NorthBridge
0000:00:18.2 Host bridge: Advanced Micro Devices [AMD] K8 NorthBridge
0000:00:18.3 Host bridge: Advanced Micro Devices [AMD] K8 NorthBridge
0000:01:08.0 Ethernet controller: Atheros Communications, Inc. AR5212 802.11abg
NIC (rev 01)
0000:01:09.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL-8139/8139C
/8139C+ (rev 10)
0000:05:00.0 VGA compatible controller: ATI Technologies Inc: Unknown device 3e5
0
0000:05:00.1 Display controller: ATI Technologies Inc: Unknown device 3e70
You might try using iperf (google it) rather than scp. There's probably some encryption overhead associated with scp, but admittedly, your performance should be better than 10Mb/s over a gigabit link.
I've got an 8169 PCI NIC, too. Later this evening I'll run some iperf tests to see what I get so we can compare.
I don't believe route tables has anything to do with my problem, but here they are...
Follows PC A routing:
Code:
root@oliaros:~# route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
192.168.3.0 * 255.255.255.0 U 0 0 0 eth0
192.168.1.0 * 255.255.255.0 U 0 0 0 eth1
default 192.168.1.1 0.0.0.0 UG 0 0 0 eth1
PC B routing is the same... (gateway is the router, eth0 is the gigabit card on both PCs)
jcliburn: I'm aware of the ssh encryption overhead, but I agree with you, 10Mb/sec on a Gbit NIC?! Thank you, I will run some tests with iperf, for comparisson.
Distribution: Slackware, CentOS. Red Hat Enterprise Linux
Posts: 216
Rep:
The issue I'm wondering about is that your performance for a 1Gb card is terrible but is right on for 100Mb card. You've got two networks (a 1Gb ans a 100Mb) and both machines can see each other directly across both networks. This may be causing some confusing in the routing and the traffic may very well be going out eth1 even though you want to go out eth0.
Using iperf, I'm getting about 437 Mb/s (average) between a VIA Velocity 6122 gigabit NIC and a Netgear GA311 (RTL-8169) gigabit NIC. There are 2 gigabit switches between the hosts. Host 192.168.1.3 (osprey) has the VIA NIC, and host 192.168.1.10 (gadwall) has the Realtek NIC.
Here's the client side.
Code:
[root@gadwall iperf-1.7.0]# ./iperf -c 192.168.1.3
------------------------------------------------------------
Client connecting to 192.168.1.3, TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[ 3] local 192.168.1.10 port 42128 connected with 192.168.1.3 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 525 MBytes 441 Mbits/sec
[root@gadwall iperf-1.7.0]# ./iperf -c 192.168.1.3
------------------------------------------------------------
Client connecting to 192.168.1.3, TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[ 3] local 192.168.1.10 port 42129 connected with 192.168.1.3 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 518 MBytes 434 Mbits/sec
[root@gadwall iperf-1.7.0]# ./iperf -c 192.168.1.3
------------------------------------------------------------
Client connecting to 192.168.1.3, TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[ 3] local 192.168.1.10 port 42130 connected with 192.168.1.3 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 518 MBytes 435 Mbits/sec
[root@gadwall iperf-1.7.0]# ./iperf -c 192.168.1.3
------------------------------------------------------------
Client connecting to 192.168.1.3, TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[ 3] local 192.168.1.10 port 42131 connected with 192.168.1.3 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 518 MBytes 435 Mbits/sec
[root@gadwall iperf-1.7.0]# ./iperf -c 192.168.1.3
------------------------------------------------------------
Client connecting to 192.168.1.3, TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[ 3] local 192.168.1.10 port 42132 connected with 192.168.1.3 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 519 MBytes 435 Mbits/sec
[root@gadwall iperf-1.7.0]#
Here's the server side.
Code:
[root@osprey iperf-1.7.0]# ./iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[ 4] local 192.168.1.3 port 5001 connected with 192.168.1.10 port 42128
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-10.0 sec 525 MBytes 441 Mbits/sec
[ 4] local 192.168.1.3 port 5001 connected with 192.168.1.10 port 42129
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-10.0 sec 518 MBytes 434 Mbits/sec
[ 4] local 192.168.1.3 port 5001 connected with 192.168.1.10 port 42130
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-10.0 sec 518 MBytes 435 Mbits/sec
[ 4] local 192.168.1.3 port 5001 connected with 192.168.1.10 port 42131
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-10.0 sec 518 MBytes 435 Mbits/sec
[ 4] local 192.168.1.3 port 5001 connected with 192.168.1.10 port 42132
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-10.0 sec 519 MBytes 436 Mbits/sec
For what it's worth, I forced my Realtek 8169 NIC to 100 Mb/s and got numbers very close to yours.
Code:
[root@gadwall iperf-1.7.0]# ethtool -s eth1 speed 100 autoneg off
[root@gadwall iperf-1.7.0]# ./iperf -c 192.168.1.3 -t 10
------------------------------------------------------------
Client connecting to 192.168.1.3, TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[ 3] local 192.168.1.10 port 45504 connected with 192.168.1.3 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 112 MBytes 94.0 Mbits/sec
This seems to point to the possibility that your NICs are autonegotiating down to 100 Mb/s. Try using ethtool to disable autonegotiation on one of the links. Chances are, your transfer won't work at all. This would imply a bad NIC or a bad cable, most likely the latter. Stupid question perhaps, but are you sure you're using a cat5e crossover? Do you have a way to validate the cable between two other systems that are known to work at gigabit speeds?
jcliburn: Thank you!!
Ok.. Latest news. NICs are working fine, problem remains..
Testing with iperf I've got 321 Mbits/sec on average which is ok. On some test I've got even >400Mbits/sec, but 312Mbits/sec is valid (for me) for a Gigabit connection.
So, what can this be, really?
Retested scp and samba after iperf tests and got the same old slow speed. (scp ~ 10MB/sec, Samba ~1,5MB/sec).
Tested hd performance again:
Code:
gecon@oliaros:~$ sudo dd if=/dev/hdb of=/dev/null count=100000
100000+0 records in
100000+0 records out
51200000 bytes transferred in 2.194738 seconds (23328525 bytes/sec)
gecon@oliaros:~$ sudo dd if=/dev/hda of=/dev/null count=100000
100000+0 records in
100000+0 records out
51200000 bytes transferred in 0.804571 seconds (63636395 bytes/sec)
Don't know why hdb is so slow (primary slave with ntfs filesystem, but isn't fs type irrelevant when using dd?) compared to hda (hdb is quite new h/w compared to hda!), but both hd's are fast enough to provide better speed to Gigabit NICs transfers (scp trasfer test with hda): hda ~ 60MB/sec, hdb ~ 22MB/sec
Any ideas?? Why are transfers so slow since iperf gives normal Gbit speeds?
1. I' ve run some tests using ftp and got about 160 Mbps speed on large file upload from PC B to PC A.
2. I' ve run ifperf -F <file> to have a file for input. As before, iperf gave good results (>320Mbps).
The next one thanks to your suggestions jcliburn:
3. There seems to be some strange behavior when I try to change settings (speed, autonegotiation) on rtl8169 on PC A (kubuntu), which is not seen when changing settings of the fast (rtl8139) card. Here it is (eth0 is the rtl8169 card):
I enter:
Code:
ethtool -s eth0 speed 100 autoneg off
and after that I run the "ethtool eth0" command.
The auto-negotiation of eth0 is reported: on. Always. Never shows 'off'.
I mean the following lines of the output:
Code:
Advertised auto-negotiation: Yes
Auto-negotiation: on
are always there, regardless of speed setting, always 'Yes', 'on'.
On my fast rtl8139 card after setting "autoneg off" the "ethtool eth1" command shows 'No', 'off' for the above fields.
Moreover the following is strange (!):
Code:
gecon@oliaros:~$ sudo ethtool -s eth0 speed 100 autoneg on
gecon@oliaros:~$ sudo ethtool eth0
Settings for eth0:
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Advertised auto-negotiation: Yes
Speed: 1000Mb/s
Duplex: Full
Port: Twisted Pair
PHYAD: 0
Transceiver: internal
Auto-negotiation: on
Link detected: yes
and after the above I run ifperf tests that gave a lot of variation. Results on many runs:
122 Mbps, 157Mbps, 395Mbps, 218Mbps, 203 Mpbs, 394Mbps, 132Mbps, 322 Mpbs
and even no connectivity, I've had to /etc/init.d/networking restart to find network.
If I enter:
Code:
gecon@oliaros:~$ sudo ethtool -s eth0 speed 100 autoneg off
gecon@oliaros:~$ sudo ethtool eth0
Settings for eth0:
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Supports auto-negotiation: Yes
Advertised link modes: 100baseT/Half 100baseT/Full
Advertised auto-negotiation: Yes
Speed: 100Mb/s
Duplex: Full
Port: Twisted Pair
PHYAD: 0
Transceiver: internal
Auto-negotiation: on
Link detected: yes
the iperf results are typical(?) for fast ethernet (82, 79, 81, 73, 91 Mpbs)
If I enter:
Code:
gecon@oliaros:~$ sudo ethtool -s eth0 speed 1000 autoneg on
gecon@oliaros:~$ sudo ethtool eth0
Settings for eth0:
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Advertised auto-negotiation: Yes
Speed: 100Mb/s
Duplex: Half
Port: Twisted Pair
PHYAD: 0
Transceiver: internal
Auto-negotiation: on
Link detected: no
the iperf results are quite good (but the problem with apps other than iperf remains). iperf gives: 333, 295,358, 357, 292, 379, 346
And finaly if I enter:
Code:
gecon@oliaros:~$ sudo ethtool -s eth0 speed 1000 autoneg off
gecon@oliaros:~$ sudo ethtool eth0
Settings for eth0:
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Supports auto-negotiation: Yes
Advertised link modes: 1000baseT/Full
Advertised auto-negotiation: Yes
Speed: 100Mb/s
Duplex: Half
Port: Twisted Pair
PHYAD: 0
Transceiver: internal
Auto-negotiation: on
Link detected: no
There is no link established.
jcliburn: I can not validate the cable as I don't have access on other Gbit devices. I bought the cable new and it says cat5e on it.
Do you think it is the cable, the NIC or maybe something else...
I' ve tested another cable which is CAT5 (not CAT5e) and it has the same behavior as the CAT5e I have which is supposed to be for Gigabit??
Ok, the questions now are:
- If the cable and the cards are ok, should a connection be established (link) always when using:
Code:
#ethtool -s eth0 speed 1000 autoneg off
?
- The fact that autoneg off is not reported as set from ethtool, suggests a software and/or hardware (NIC) problem?
- How come the ifperf report such good speeds (over 100Mbps difference) compared to every other traffic?
Just a quick tip about gigabit ethernet. In gigabit ethernet the transfer uses all 4 pair (not like 10/100 1,2,3 and 6) so if you make a crossover cable for gig, i assume you changed pins 1,2,3, and 6 causing the card to stay at 100. There might be a way to roll the cable but in all my gigabit network a device is needed to get full through-put. cat5 cable is good for gig also.
dclark: The cable is CAT5E bought from a store (I have not made it).
Gigabit ethernet NICs can operate and establish a link with plain cable (not crossed over) even in direct connection (peer 2 peer) between the NICs.
I only have 1 CAT5e Cable here (the one which states "crossover").
I also have tried 2 different CAT5 cables (no crossover) which gave the same behavior as the CAT5e crossover cable.
My results are similar to yours whenever I dork with the 8169 settings. As a matter of fact, I need to walk upstairs and reset autoneg back to "on" after I lost the link by setting speed=1000 and autoneg=off.
I think you've got a good NIC and a good cable. scp performance simply sucks, I guess, owing to the encryption. I see the same thing transferring a file full of floating point numbers I ginned up.
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.