Fake fullduplex

Pearlseattle · 06-21-2011, 07:26 AM

Hi
Yesterday, after a long debugging session, I came to the conclusion that my network at home is not running in full duplex.

The network I have looks the following:
PC1 <=> SWITCH <=> PC2

- I am using at least cat.5 cables.
- On all PCs dmesg, ethtool, mii-something, etc... show that the cards of both PCs are in full duplex.
- The switch is happy (all the lights of the switch http://www.alliedtelesis.com/media/d...0_16ECO_DS.pdf are green).
- Transferring data 1-way works perfectly (~100MB/s).
- Transferring data 2-ways makes me always reach not more than a total of ~100-120MB/s.
- Tested all this using 4 PCs, in different configurations.
- Tested using 2 different Gbit switches (both supposed to be fullduplex).
- Until now I always ran test having one of the switches inbetween. I still have to run a test having the two PC connected directly.
- The CPUs on the PCs are powerful enough (quad-cores in both cases).

To run the tests I used, apart from real file transfers (using glusterfs) especially the utility "iperf" with the parameter "-d" to run the test in both directions simultaneously.

I'm not sure if I'm just missing something very simple.
How does it look for you? If you start the iperf server on one of the PCs (with "iperf -s") and run the client on another PC (with "iperf -c <targetserver> -d"), and you servers are connected through a Gbit network, do you see 1Gbit/s in both directions?

Maaany thanks!!

Pearlseattle · 06-21-2011, 11:21 AM

Mh, just ran a test with 2 PCs connected directly - no switch involved.

Code:

PC1 ~ # mii-tool 
eth1: negotiated 1000baseT-FD flow-control, link ok

PC2 ~ # mii-tool 
eth0: negotiated 1000baseT-FD flow-control, link ok

So, both are 1Gbit FD (full duplex).

Test, first only in one direction, then concurrently in both (have already started "iperf -s" on PC2).

Code:

PC1 ~ # iperf -c 10.0.4.2
------------------------------------------------------------
Client connecting to 10.0.4.2, TCP port 5001
TCP window size: 1.00 MByte (default)
------------------------------------------------------------
[  3] local 10.0.4.1 port 57805 connected with 10.0.4.2 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec   783 MBytes   657 Mbits/sec

PC1 ~ # iperf -c 10.0.4.2 -d
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 1.00 MByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to 10.0.4.2, TCP port 5001
TCP window size: 1.00 MByte (default)
------------------------------------------------------------
[  5] local 10.0.4.1 port 57806 connected with 10.0.4.2 port 5001
[  4] local 10.0.4.1 port 5001 connected with 10.0.4.2 port 45461
[ ID] Interval       Transfer     Bandwidth
[  5]  0.0-10.0 sec   611 MBytes   512 Mbits/sec
[  4]  0.0-10.0 sec   180 MBytes   151 Mbits/sec

Shouldn't the transfer rates for both directions be of ~657 Mbit/s, instead of 512 and 151?

Ifconfig says:

Code:

PC1 ~ # ifconfig eth1
eth1      Link encap:Ethernet  HWaddr 00:0f:b5:fb:ba:8c  
          inet addr:10.0.4.1  Bcast:10.0.4.255  Mask:255.255.255.0
          inet6 addr: fe80::20f:b5ff:fefb:ba8c/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:633097 errors:0 dropped:183 overruns:0 frame:0
          TX packets:1069360 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:230597826 (219.9 MiB)  TX bytes:1532585570 (1.4 GiB)
          Interrupt:16 Base address:0xec00

(that "dropped: 183" is a bit weird...)

Code:

PC2 ~ # ifconfig eth0
eth0      Link encap:Ethernet  HWaddr 00:15:17:31:a3:42  
          inet addr:10.0.4.2  Bcast:10.0.4.255  Mask:255.255.255.0
          inet6 addr: fe80::215:17ff:fe31:a342/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1069325 errors:0 dropped:0 overruns:0 frame:0
          TX packets:633251 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:1532580524 (1.4 GiB)  TX bytes:231252412 (220.5 MiB)
          Memory:ff6e0000-ff700000

Don't know what I should do...

Pearlseattle · 06-21-2011, 11:28 AM

p.s.: used a cat.6 cable

jefro · 06-21-2011, 03:38 PM

You ought to just edit your posts. It looks like many people have answered you when you post over and over.

They are in full dux.

I suspect the system can't support anywhere near gig speeds. Too much system overhead and lack of speed on it's end. Nic is fine.

Pearlseattle · 08-05-2011, 02:57 PM

Quote:

I suspect the system can't support anywhere near gig speeds. Too much system overhead and lack of speed on it's end. Nic is fine.

Mmmh, you were partially right - I didn't post infos about my HW, so you couldn't exclude that.
In any case the PCs I used to perform the tests were both quad-cores Intel Q9550 & Q6600 CPUs, both with 4GB RAM, so I excluded that the problem was generated by the system itself.
BUT: I changed the NICs and went from Atheros (Motherboard) (first attempt) and Realtek (PCI) (second attempt) to Intel 82574L (PCI-E 1x) and I now get real fullduplex (don't remember exactly - if I remember correctly it's ~800MBit/s in both directions through a switch).

The Intel NIC even uses multiple IRQs (but only with the appropriate kernel settings) - wow:

Code:

MYSERVER~ # cat /proc/interrupts | egrep "CPU|eth4"
           CPU0       CPU1       CPU2       CPU3       
 46:       3233       3346       3230       3269   PCI-MSI-edge      eth4-rx-0
 47:       3303       3249       3432       3232   PCI-MSI-edge      eth4-tx-0
 48:          1          0          1          1   PCI-MSI-edge      eth4

Happy!

Thx