gigabit ethernet really slow

g_k · 07-06-2007, 02:19 PM

Hi!

The performance of my Gigabit Ethernet is really bad. I have 2 hosts with up-to-date gentoo installations, three kinds of gigabit ethernet cards (Marvell Yukon - skge, Realtek - r8169, Intel - e1000) a gigabit switch (3com) and 2 different cables (patch and crossover, both Cat6). I tried all possible combinations between the cards with and without the switch, however, when I scp a file from one host to another I get not more than ~19MB/s. When using 100MBit cards I get 11MB/s!

I'm really stuck now...

Any help greatly appreciated!

Thx, georg

P.S.:

The current setup is "host1(e1000) <-> switch <-> host2(skge)":
ethtool output on host1:

Code:

Supported ports: [ TP ]
        Supported link modes:   10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Supports auto-negotiation: Yes
        Advertised link modes:  10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Advertised auto-negotiation: Yes
        Speed: 1000Mb/s
        Duplex: Full
        Port: Twisted Pair
        PHYAD: 0
        Transceiver: internal
        Auto-negotiation: on
        Supports Wake-on: umbg
        Wake-on: d
        Current message level: 0x00000007 (7)
        Link detected: yes

ethtool output on host2:

Code:

Settings for eth0:
        Supported ports: [ TP ]
        Supported link modes:   10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Supports auto-negotiation: Yes
        Advertised link modes:  10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Advertised auto-negotiation: Yes
        Speed: 1000Mb/s
        Duplex: Full
        Port: Twisted Pair
        PHYAD: 0
        Transceiver: internal
        Auto-negotiation: on
        Supports Wake-on: g
        Wake-on: d
        Current message level: 0x00000037 (55)
        Link detected: yes

t1n0m3n · 07-06-2007, 02:33 PM

Does ifconfig show errors?

Are you sure that you are not bottle-necking somewhere else.... Like the write capacity of the hard drive?

My home network could be gig, but since I have a laptop it is worthless (for what I would use it for). I can't go much beyond 100Mbps transfers anyway. My laptop can't write to HD that fast.

g_k · 07-08-2007, 06:27 AM

host1:

Code:

$ dd if=/dev/zero of=testfile bs=16k count=196608
196608+0 Datensätze ein
196608+0 Datensätze aus
3221225472 Bytes (3,2 GB) kopiert, 58,2955 s, 55,3 MB/s

ifconfig:
  inet Adresse:192.168.1.1  Bcast:192.168.1.255  Maske:255.255.255.0
  UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
  RX packets:2277332 errors:0 dropped:0 overruns:0 frame:0
  TX packets:5445228 errors:0 dropped:0 overruns:0 carrier:0
  Kollisionen:0 Sendewarteschlangenlänge:1000
  RX bytes:165588568 (157.9 Mb)  TX bytes:3559215059 (3394.3 Mb)
  Basisadresse:0xa000 Speicher:da800000-da820000

host2:

Code:

$ dd if=/dev/zero of=testfile bs=16k count=196608
196608+0 records in
196608+0 records out
3221225472 bytes (3.2 GB) copied, 71.6973 s, 44.9 MB/s

ifconfig:
  inet addr:192.168.1.5  Bcast:192.168.1.255  Mask:255.255.255.0
  UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
  RX packets:5443044 errors:0 dropped:0 overruns:0 frame:0
  TX packets:2275232 errors:0 dropped:0 overruns:0 carrier:0
  collisions:0 txqueuelen:1000
  RX bytes:3558945484 (3394.0 Mb)  TX bytes:163996442 (156.3 Mb)
  Interrupt:17

Where else could be a bottle neck?

Thx, georg

t1n0m3n · 07-08-2007, 08:33 PM

What am I looking at here?
Those throughput times look like read *not write* times to me.
Your write to hard drive throughput should be much slower. (Unless these are RAID setups of course)

When transferring files, I take it as a general rule of thumb that I am limited by the write speed of the hard drive of the machine that I am transferring to.

I have much better throughput when transferring files *from* my laptop than *to* my laptop.

You could also be running into CPU issues as well. Although my money is on Hard Drive is your bottle neck.

*EDIT* BTW, I have 2 raptors in Raid 0 on my home machine, so it is a lot more obvious to me when I transfer files back and forth. Although I have yet to check if my home machine's write speed is actually faster than my laptop's read speed.

t1n0m3n · 07-08-2007, 08:43 PM

Some other thoughts....

If you were to mount part of your ram as a drive, I wonder what the result would be. (Transferring from ram drive to ram drive that is)

Also, if you were to disclose what your HD setups on both machines were, that may shed some light on this.

g_k · 07-09-2007, 12:14 PM

Quote:

What am I looking at here?
Those throughput times look like read *not write* times to me.

I really just copy-pasted from console, so they are write times.
Host1 has a standard IDE hard drive I bought last year, and on host2 there is a software raid5 with 3 disks.

However, suddenly it struck me that scp cannot be a good benchmark because it encrypts the data and maybe also adds some other overhead. So I did some tests with dd on the mounted NFS share:
read: ~50MB/s
write:~15MB/s

I still don't understand the bad write performance (I double-checked those local 45MB/s), however, I can live with this situation now...

Thx for your help! Regards, georg

thorn168 · 07-09-2007, 06:09 PM

How long are the cables? Have they been properly terminated? Why are you using a crossover cable? (what is its purpose in the network scheme?)

As for hardware...how fast is your PCI bus? I would guess that the PCI bus is the source of the bottleneck.

Network-->NIC-->PCI bus-->CPU-->HD

If the network and the Nic are fast and the CPU and the HD are fast, the only thing that could slow them all down is the PCI bus.

michaelnel · 07-09-2007, 07:17 PM

Did you try jumbo frames? The nics and the switch would have to both be set for jumbo (mtu=9000)

t1n0m3n · 07-13-2007, 03:57 PM

When I said "Those throughput times look like read *not write* times to me." I meant that the numbers themselves look like read times.
It also seemed odd that your host1's single drive is writing faster than your raid array. Could be to the newer drive technology.

So a couple of assumptions:

1. Transfers from host1 to host2 should be faster due to host2 being able to write to its raid drive at a faster rate.
2. Transfers from host2 to host1 should bottleneck on the 15MB/s write speed of host1's standard IDE HD.

A comparison of these two assumption scenarios plus an IPerf benchmark (in both directions) might be interesting.

The only other thing I can think of is a bad NIC. I have seen a NIC give asymmetric throughput results before.

thorn168: You are assuming that these NICs are actually on a PCI bus.
g_k: what kind of NICs are in these machines and if they are built-in, what are the makes and models of the motherboards?

Abida54 · 04-07-2010, 05:49 PM

According to my experience Ethernet throughput is highly dependent on the size of the packets sent. maybe the files you are copying are small. I had the same problem a while ago.

here is the link that shows how the throughput is dependent on the size of the packets:

http://support.novell.com/techcenter...a19960103.html

jefro · 04-07-2010, 08:15 PM

Might peek here too.

http://www.enterpriseitplanet.com/ne...le.php/3497796

Almost the same.

http://www.enterprisenetworkingplane...0950_3485486_2