LinuxQuestions.org
Review your favorite Linux distribution.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Networking
User Name
Password
Linux - Networking This forum is for any issue related to networks or networking.
Routing, network cards, OSI, etc. Anything is fair game.

Notices


Reply
  Search this Thread
Old 04-06-2008, 09:21 AM   #1
DaveQB
Member
 
Registered: Oct 2003
Location: Sydney, Australia.
Distribution: PCLinuxOS 2010.12, Debian Lenny
Posts: 396

Rep: Reputation: 38
[SOLVED] Ethernet gigabit NFS performance poor


I have layed out Cat6 cable, got a Gbit NIC in my desktop, myth box and server. Using 2 Gigabit switches now.

So I expected to see some good network throughput, but no! :-(

When copying a good size file (500MB >) I have noticed is goes in bursts. So I can transfer at 50MB/sec, which I am very happy with, then after about 200MB it slows to a crawl and even stops. Then, it will burst back up to 50MB/sec or so.

On average, files are copying at 9.5MB/second.

Any tips to work out why ?

Oh the NFS CPU's is hardly working at all, something I have noticed.

Last edited by DaveQB; 02-03-2011 at 06:30 PM. Reason: Marked solved
 
Old 04-06-2008, 01:14 PM   #2
MensaWater
LQ Guru
 
Registered: May 2005
Location: Atlanta Georgia USA
Distribution: Redhat (RHEL), CentOS, Fedora, CoreOS, Debian, FreeBSD, HP-UX, Solaris, SCO
Posts: 7,829
Blog Entries: 15

Rep: Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668
Run ethtool against both interfaces to see what settings you see.

Just run "ethtool eth<#> where # is the interface you're using (e.g. eth0).

It should produce results similar to:
Settings for eth0:
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Supports auto-negotiation: Yes
Advertised link modes: Not reported
Advertised auto-negotiation: No
Speed: 100Mb/s
Duplex: Full
Port: Twisted Pair
PHYAD: 1
Transceiver: internal
Auto-negotiation: off
Supports Wake-on: g
Wake-on: d
Link detected: yes

If the card and the switch are both Gigabit you want "Advertised auto-negotiation" to be "Yes" rathern than the "No" in above. You also want the speed to show 1000 rather than 100 seen above.

Also you need to make the switch agree.

If your switch port or card only does 100Mb you do want "Advertised auto-negotiation" to be "No" and you want to forced the speed to 100Mb/s and duplex to full. It is very common when you have 100Mb that auto-negotiation forces it to half-duplex which slows things down to a crawl. Setting the card to 100/Full with no auto-negotiation will help. (With Gigabit you MUST do auto-negotiation to get 1000Mb/s.) The ethtool command can be used to set things on the cards.

I believe on the switch itself you CAN force it to 1000Mb/s Full so setting auto-negotiation on the card has the effect of insuring it gets 1000Mb/s Full. (Assuming the card is capable of that.)

You can find out how to set cards by typing "man ethtool".

FYI: On some older cards ethtool doesn't have any effect so using mii-tool instead may be necessary. It too has a man page.
 
Old 04-06-2008, 03:28 PM   #3
lazlow
Senior Member
 
Registered: Jan 2006
Posts: 4,363

Rep: Reputation: 172Reputation: 172
I would disagree with "With Gigabit you MUST do auto-negotiation to get 1000Mb/s". Mine works fine with auto negotiate off.


I would guess that you are dropping back to 100. The 9.5MB/sec is about as good as it gets on 100.

You also want to make sure that your drives on both ends can handle the throughput " hdparm -tT /dev/sda".
 
Old 04-06-2008, 04:45 PM   #4
DaveQB
Member
 
Registered: Oct 2003
Location: Sydney, Australia.
Distribution: PCLinuxOS 2010.12, Debian Lenny
Posts: 396

Original Poster
Rep: Reputation: 38
Hi,

Thanks for the replies you 2.
Yes, the switch is showing the different colour for the ports both of these PC's are hooked into, indicating it running at Gigabit.

First part.

My Desktop

Code:
david@david ~ $ sudo ethtool  eth0
[sudo] password for david:
Settings for eth0:
        Supported ports: [ MII ]
        Supported link modes:   10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Supports auto-negotiation: Yes
        Advertised link modes:  10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Advertised auto-negotiation: Yes
        Speed: 1000Mb/s
        Duplex: Full
        Port: MII
        PHYAD: 0
        Transceiver: external
        Auto-negotiation: on
        Supports Wake-on: g
        Wake-on: d
        Link detected: yes
Myth Box

Code:
david@myth:~$ sudo ethtool  eth0
Password:
Settings for eth0:
        Supported ports: [ MII ]
        Supported link modes:   10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Supports auto-negotiation: Yes
        Advertised link modes:  10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Advertised auto-negotiation: Yes
        Speed: 1000Mb/s
        Duplex: Full
        Port: MII
        PHYAD: 1
        Transceiver: external
        Auto-negotiation: on
        Supports Wake-on: g
        Wake-on: d
        Link detected: yes
Hard disk performance.

On my desktop

Code:
david@david ~ $ sudo hdparm -tT /dev/sdd2

/dev/sdd2:
 Timing cached reads:   2002 MB in  2.00 seconds = 1000.03 MB/sec
 Timing buffered disk reads:  220 MB in  3.00 seconds =  73.26 MB/sec

On the myth box

Code:
david@myth:~$ sudo hdparm -tT /dev/hda2

/dev/hda2:
 Timing cached reads:   1482 MB in  2.00 seconds = 740.90 MB/sec
 Timing buffered disk reads:  178 MB in  3.03 seconds =  58.74 MB/sec
 
Old 04-06-2008, 05:15 PM   #5
lazlow
Senior Member
 
Registered: Jan 2006
Posts: 4,363

Rep: Reputation: 172Reputation: 172
Well the drives are not the problem.

I would check the ethtool again during a large file transfer when it seems to slow. It MAY be dropping back during the transfer. It has been a while but I THINK that is why I have my auto-negotiation set to off.
 
Old 04-06-2008, 06:08 PM   #6
MensaWater
LQ Guru
 
Registered: May 2005
Location: Atlanta Georgia USA
Distribution: Redhat (RHEL), CentOS, Fedora, CoreOS, Debian, FreeBSD, HP-UX, Solaris, SCO
Posts: 7,829
Blog Entries: 15

Rep: Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668
Not disk, not network, not CPU (from your first post) which leaves memory. Run the free command and vmstat for more details on memory utilization.

Also are you doing any sort of encryption in the file transfer you mentioned? Encryption can slow down file transfers a great deal. I assumed from your original post that you mean you were simply doing a copy from one filesystem to the NFS mounted filesystem on the same box.

Finally you might want to look at setting MTU value. I've seen issues in the past where the MTU on the switch needed to be adjusted but can't really give much guidance there as that is something the Network Admins typically do where I work.
 
Old 04-07-2008, 12:46 AM   #7
DaveQB
Member
 
Registered: Oct 2003
Location: Sydney, Australia.
Distribution: PCLinuxOS 2010.12, Debian Lenny
Posts: 396

Original Poster
Rep: Reputation: 38
Quote:
Originally Posted by lazlow View Post
Well the drives are not the problem.

I would check the ethtool again during a large file transfer when it seems to slow. It MAY be dropping back during the transfer. It has been a while but I THINK that is why I have my auto-negotiation set to off.
Thanks

So when you say check with ethtool, just check what mode it is in ? 100 or gigabit ?
 
Old 04-07-2008, 12:49 AM   #8
DaveQB
Member
 
Registered: Oct 2003
Location: Sydney, Australia.
Distribution: PCLinuxOS 2010.12, Debian Lenny
Posts: 396

Original Poster
Rep: Reputation: 38
Quote:
Originally Posted by jlightner View Post
Not disk, not network, not CPU (from your first post) which leaves memory. Run the free command and vmstat for more details on memory utilization.

Also are you doing any sort of encryption in the file transfer you mentioned? Encryption can slow down file transfers a great deal. I assumed from your original post that you mean you were simply doing a copy from one filesystem to the NFS mounted filesystem on the same box.

Finally you might want to look at setting MTU value. I've seen issues in the past where the MTU on the switch needed to be adjusted but can't really give much guidance there as that is something the Network Admins typically do where I work.
I'll check memory. I have 2GB (formerly 4gB) on my desktop and 1GB in the myth box (same size as more recent file I copied (1.2GB) and had trouble with (9.5MB/sec))

No encryption, copying to a mount NFS share, thats it.

I cant change MTU on switch, its an unmanaged switch. For home use. Gigabit.
 
Old 04-07-2008, 12:50 AM   #9
lazlow
Senior Member
 
Registered: Jan 2006
Posts: 4,363

Rep: Reputation: 172Reputation: 172
Yep, that 9.5 speed looks a lot like a maxed out 100 rather than GigE. Something MAY be deciding (when under heavy load) that it should shift down to 100 rather than GigE. If that is what is going on the only way to find out is to check while it is working under those conditions.


My current media box is a 1Ghz PIII with 512 ram and it transfers fast enough to keep the HD max (around 50).

Last edited by lazlow; 04-07-2008 at 12:52 AM.
 
Old 04-07-2008, 12:55 AM   #10
DaveQB
Member
 
Registered: Oct 2003
Location: Sydney, Australia.
Distribution: PCLinuxOS 2010.12, Debian Lenny
Posts: 396

Original Poster
Rep: Reputation: 38
Quote:
Originally Posted by lazlow View Post
Yep, that 9.5 speed looks a lot like a maxed out 100 rather than GigE. Something MAY be deciding (when under heavy load) that it should shift down to 100 rather than GigE. If that is what is going on the only way to find out is to check while it is working under those conditions.
It does burst up around 50MB/sec and over, then slows to a halt. then about 5-10 seconds later, flies back up to 50MB/sec plus. and the process continues.

My network doesn't do this, I dont think, when copying to a 10/100 NIC PC. Its more consistent. This is using Konqueror.



Quote:
Originally Posted by lazlow View Post
My current media box is a 1Ghz PIII with 512 ram and it transfers fast enough to keep the HD max (around 50).


I am jealous!!!!


Thanks for responding.
 
Old 04-08-2008, 02:23 AM   #11
DaveQB
Member
 
Registered: Oct 2003
Location: Sydney, Australia.
Distribution: PCLinuxOS 2010.12, Debian Lenny
Posts: 396

Original Poster
Rep: Reputation: 38
Quote:
Originally Posted by lazlow View Post
Yep, that 9.5 speed looks a lot like a maxed out 100 rather than GigE. Something MAY be deciding (when under heavy load) that it should shift down to 100 rather than GigE. If that is what is going on the only way to find out is to check while it is working under those conditions.


My current media box is a 1Ghz PIII with 512 ram and it transfers fast enough to keep the HD max (around 50).
Well just tried a 700MB file. free showed hardly any impact on memory usage on both server and client during the transfer.

Also, I double checked CPU usage with top and the client showed 40% or so for wa, so that's the network slowing it. On the server side, nfsd had about 4 process up the top of the chart, but they were all around 2-3%.
So nothing there.

Ethtool showed the same as above for both machines through out.

So I will try again with auto-negotiation turned off. Which do you turn off ? Supported auto-negotiation or Advertised auto-negotiation ?
 
Old 04-09-2008, 01:41 PM   #12
MensaWater
LQ Guru
 
Registered: May 2005
Location: Atlanta Georgia USA
Distribution: Redhat (RHEL), CentOS, Fedora, CoreOS, Debian, FreeBSD, HP-UX, Solaris, SCO
Posts: 7,829
Blog Entries: 15

Rep: Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668
Advertised.

Supported tells you which levels it can autonegotiate to if it is advertised.
 
Old 04-19-2008, 09:11 AM   #13
DaveQB
Member
 
Registered: Oct 2003
Location: Sydney, Australia.
Distribution: PCLinuxOS 2010.12, Debian Lenny
Posts: 396

Original Poster
Rep: Reputation: 38
Am I missing something here ?

Code:
david@myth:~$ sudo ethtool  -s eth0 autoneg off
Cannot set new settings: Invalid argument
  not setting autoneg

Both boxes do this ...
 
Old 04-19-2008, 09:20 AM   #14
MensaWater
LQ Guru
 
Registered: May 2005
Location: Atlanta Georgia USA
Distribution: Redhat (RHEL), CentOS, Fedora, CoreOS, Debian, FreeBSD, HP-UX, Solaris, SCO
Posts: 7,829
Blog Entries: 15

Rep: Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668
I suspect it is because your card doesn't support autonegotiate off at speed 1000 (as I originally stated and was contradicted on).

Try setting speed to 100 and autonegotiate off OR leave speed at 1000 with autonegotiate on and hardcode the switch port to 1000 with autonegotiate off.
 
Old 04-21-2008, 11:33 PM   #15
DaveQB
Member
 
Registered: Oct 2003
Location: Sydney, Australia.
Distribution: PCLinuxOS 2010.12, Debian Lenny
Posts: 396

Original Poster
Rep: Reputation: 38
Quote:
Originally Posted by jlightner View Post
Try setting speed to 100 and autonegotiate off OR leave speed at 1000 with autonegotiate on and hardcode the switch port to 1000 with autonegotiate off.

Set speed to 100 ? But then that defeats the purpose of what I am trying to achieve, right ?

How do you mean "hardcode the switch port to 1000..." ???

Thanks
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
LXer: Optimize high-performance Linux and gigabit ethernet LXer Syndicated Linux News 0 01-12-2008 12:00 PM
Switch Hell...NFS and degrading performance with gigabit switched edman007 Linux - Networking 4 03-12-2006 06:49 PM
poor performance javabb Red Hat 8 01-20-2005 05:48 AM
poor performance in x ah786 Slackware 10 11-09-2004 11:11 AM
Poor performance Shyne Red Hat 1 10-17-2004 11:50 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Networking

All times are GMT -5. The time now is 02:00 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration