poor NFS performance for one client only
Hello,
I have a real head scratcher here. I have three application servers that all use three volumes mounted off of two NFS servers. All operating systems involved are RHEL 6.3. dss01a - application dss01b - application dss01c - application nfs01a - NFS server exports: /data01b nfs01b - NFS server exports: /data01a, data01c The problem is dss01c gets around 5 MB/s basic write speed to nfs01b (both /data01a and /data01c volumes) over NFS. This server gets over 100 MB/s to the other NFS server. The problem does not occur over other protocols such as SSH or using netcat. The configuration between the three application servers is identical (NICs, mount, etc). The configuration between the two NFS servers is also identical. Since I am not seeing any issues with any other protocol, I am ruling out network configuration and thus purely focusing on NFS. Does anyone have any additional troubleshooting ideas to help me narrow this down? |
Id like to see "ifconfig -a" output of all clients and the nfs01b. And "/sbin/mii-tool -v" if you have it. Are the clients all on the same package versions?
|
First, thank you for the reply. mii-tool shows they negotiated at 100/Full but ethtool shows 1000/Full.
dss01c (problem client) [root@dss01c ~]# ifconfig -a bond0 Link encap:Ethernet HWaddr 00:19:B9:E7:02:53 inet addr:10.199.1.102 Bcast:10.199.1.255 Mask:255.255.255.0 inet6 addr: fe80::219:b9ff:fee7:253/64 Scope:Link UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1 RX packets:13029042 errors:0 dropped:0 overruns:0 frame:0 TX packets:9811625 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:8616213029 (8.0 GiB) TX bytes:5599520278 (5.2 GiB) bond1 Link encap:Ethernet HWaddr 00:15:17:AD:22:32 inet addr:10.198.1.102 Bcast:10.198.1.255 Mask:255.255.255.0 inet6 addr: fe80::215:17ff:fead:2232/64 Scope:Link UP BROADCAST RUNNING MASTER MULTICAST MTU:9000 Metric:1 RX packets:26450069 errors:0 dropped:0 overruns:0 frame:0 TX packets:24731532 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:134022024904 (124.8 GiB) TX bytes:109700344753 (102.1 GiB) eth0 Link encap:Ethernet HWaddr 00:19:B9:E7:02:53 UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1 RX packets:1501842 errors:0 dropped:0 overruns:0 frame:0 TX packets:406819 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:2000 RX bytes:494706070 (471.7 MiB) TX bytes:57499228 (54.8 MiB) Interrupt:16 Memory:f8000000-f8012800 eth1 Link encap:Ethernet HWaddr 00:19:B9:E7:02:53 UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1 RX packets:11527200 errors:0 dropped:0 overruns:0 frame:0 TX packets:9404806 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:2000 RX bytes:8121506959 (7.5 GiB) TX bytes:5542021050 (5.1 GiB) Interrupt:16 Memory:f4000000-f4012800 lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:33 errors:0 dropped:0 overruns:0 frame:0 TX packets:33 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:5341 (5.2 KiB) TX bytes:5341 (5.2 KiB) p2p1 Link encap:Ethernet HWaddr 00:15:17:AD:22:32 UP BROADCAST RUNNING SLAVE MULTICAST MTU:9000 Metric:1 RX packets:7049557 errors:0 dropped:0 overruns:0 frame:0 TX packets:10164437 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:2000 RX bytes:8203494848 (7.6 GiB) TX bytes:22263200771 (20.7 GiB) Interrupt:16 Memory:fc3e0000-fc400000 p2p2 Link encap:Ethernet HWaddr 00:15:17:AD:22:32 UP BROADCAST RUNNING SLAVE MULTICAST MTU:9000 Metric:1 RX packets:19400512 errors:0 dropped:0 overruns:0 frame:0 TX packets:14567095 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:2000 RX bytes:125818530056 (117.1 GiB) TX bytes:87437143982 (81.4 GiB) [root@dss01c ~]# cat /proc/net/bonding/bond1 Ethernet Channel Bonding Driver: v3.6.0 (September 26, 2009) Bonding Mode: IEEE 802.3ad Dynamic link aggregation Transmit Hash Policy: layer2 (0) MII Status: up MII Polling Interval (ms): 60 Up Delay (ms): 0 Down Delay (ms): 0 802.3ad info LACP rate: slow Aggregator selection policy (ad_select): stable Active Aggregator Info: Aggregator ID: 2 Number of ports: 2 Actor Key: 17 Partner Key: 10013 Partner Mac Address: 00:1b:ed:80:17:c0 Slave Interface: p2p1 MII Status: up Speed: 1000 Mbps Duplex: full Link Failure Count: 1 Permanent HW addr: 00:15:17:ad:22:32 Aggregator ID: 2 Slave queue ID: 0 Slave Interface: p2p2 MII Status: up Speed: 1000 Mbps Duplex: full Link Failure Count: 1 Permanent HW addr: 00:15:17:ad:22:33 Aggregator ID: 2 Slave queue ID: 0 [root@dss01c ~]# mii-tool -v p2p1 p2p1: negotiated 100baseTx-FD flow-control, link ok product info: vendor 00:aa:00, model 56 rev 0 basic mode: autonegotiation enabled basic status: autonegotiation complete, link ok capabilities: 100baseTx-FD 100baseTx-HD 10baseT-FD 10baseT-HD advertising: 100baseTx-FD 100baseTx-HD 10baseT-FD 10baseT-HD flow-control link partner: 100baseTx-FD 100baseTx-HD 10baseT-FD 10baseT-HD flow-control [root@dss01c ~]# mii-tool -v p2p2 p2p2: negotiated 100baseTx-FD flow-control, link ok product info: vendor 00:aa:00, model 56 rev 0 basic mode: autonegotiation enabled basic status: autonegotiation complete, link ok capabilities: 100baseTx-FD 100baseTx-HD 10baseT-FD 10baseT-HD advertising: 100baseTx-FD 100baseTx-HD 10baseT-FD 10baseT-HD flow-control link partner: 100baseTx-FD 100baseTx-HD 10baseT-FD 10baseT-HD flow-control dss01b (good client) [root@dss01b ~]# ifconfig -a bond0 Link encap:Ethernet HWaddr 00:1D:09:71:98:CA inet addr:10.199.1.101 Bcast:10.199.1.255 Mask:255.255.255.0 inet6 addr: fe80::21d:9ff:fe71:98ca/64 Scope:Link UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1 RX packets:2956041533 errors:0 dropped:0 overruns:0 frame:0 TX packets:3952410818 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:2482716743909 (2.2 TiB) TX bytes:4187156004173 (3.8 TiB) bond1 Link encap:Ethernet HWaddr 00:15:17:A5:66:28 inet addr:10.198.1.101 Bcast:10.198.1.255 Mask:255.255.255.0 inet6 addr: fe80::215:17ff:fea5:6628/64 Scope:Link UP BROADCAST RUNNING MASTER MULTICAST MTU:9000 Metric:1 RX packets:6393965280 errors:0 dropped:0 overruns:0 frame:0 TX packets:5924477304 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:31398320705419 (28.5 TiB) TX bytes:5886681482828 (5.3 TiB) eth0 Link encap:Ethernet HWaddr 00:15:17:A5:66:28 UP BROADCAST RUNNING SLAVE MULTICAST MTU:9000 Metric:1 RX packets:3049351898 errors:0 dropped:0 overruns:0 frame:0 TX packets:394198413 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:2000 RX bytes:15942999947136 (14.5 TiB) TX bytes:304387170969 (283.4 GiB) Interrupt:16 Memory:fc3e0000-fc400000 eth1 Link encap:Ethernet HWaddr 00:15:17:A5:66:28 UP BROADCAST RUNNING SLAVE MULTICAST MTU:9000 Metric:1 RX packets:3344613382 errors:0 dropped:0 overruns:0 frame:0 TX packets:5530278891 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:2000 RX bytes:15455320758283 (14.0 TiB) TX bytes:5582294311859 (5.0 TiB) Interrupt:17 Memory:fc3a0000-fc3c0000 eth2 Link encap:Ethernet HWaddr 00:1D:09:71:98:CA UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1 RX packets:1469795632 errors:0 dropped:0 overruns:0 frame:0 TX packets:3926237389 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:2000 RX bytes:1226697697626 (1.1 TiB) TX bytes:4169371617353 (3.7 TiB) Interrupt:16 Memory:f8000000-f8012800 eth3 Link encap:Ethernet HWaddr 00:1D:09:71:98:CA UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1 RX packets:1486245901 errors:0 dropped:0 overruns:0 frame:0 TX packets:26173429 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:2000 RX bytes:1256019046283 (1.1 TiB) TX bytes:17784386820 (16.5 GiB) Interrupt:16 Memory:f4000000-f4012800 lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:5052891 errors:0 dropped:0 overruns:0 frame:0 TX packets:5052891 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:423367352 (403.7 MiB) TX bytes:423367352 (403.7 MiB) [root@dss01b ~]# cat /proc/net/bonding/bond1 Ethernet Channel Bonding Driver: v3.6.0 (September 26, 2009) Bonding Mode: IEEE 802.3ad Dynamic link aggregation Transmit Hash Policy: layer2 (0) MII Status: up MII Polling Interval (ms): 60 Up Delay (ms): 0 Down Delay (ms): 0 802.3ad info LACP rate: slow Aggregator selection policy (ad_select): stable Active Aggregator Info: Aggregator ID: 1 Number of ports: 2 Actor Key: 17 Partner Key: 10006 Partner Mac Address: 00:1b:ed:80:17:c0 Slave Interface: eth0 MII Status: up Speed: 1000 Mbps Duplex: full Link Failure Count: 2 Permanent HW addr: 00:15:17:a5:66:28 Aggregator ID: 1 Slave queue ID: 0 Slave Interface: eth1 MII Status: up Speed: 1000 Mbps Duplex: full Link Failure Count: 2 Permanent HW addr: 00:15:17:a5:66:29 Aggregator ID: 1 Slave queue ID: 0 [root@dss01b ~]# mii-tool -v eth0 eth0: negotiated 100baseTx-FD flow-control, link ok product info: vendor 00:aa:00, model 56 rev 0 basic mode: autonegotiation enabled basic status: autonegotiation complete, link ok capabilities: 100baseTx-FD 100baseTx-HD 10baseT-FD 10baseT-HD advertising: 100baseTx-FD 100baseTx-HD 10baseT-FD 10baseT-HD flow-control link partner: 100baseTx-FD 100baseTx-HD 10baseT-FD 10baseT-HD flow-control [root@dss01b ~]# mii-tool -v eth1 eth1: negotiated 100baseTx-FD flow-control, link ok product info: vendor 00:aa:00, model 56 rev 0 basic mode: autonegotiation enabled basic status: autonegotiation complete, link ok capabilities: 100baseTx-FD 100baseTx-HD 10baseT-FD 10baseT-HD advertising: 100baseTx-FD 100baseTx-HD 10baseT-FD 10baseT-HD flow-control link partner: 100baseTx-FD 100baseTx-HD 10baseT-FD 10baseT-HD flow-control For good measure, here's one of the ethtool outputs that shows we are at 1000/Full. I'll triple check on the switch side too: [root@dss01b ~]# ethtool eth0 Settings for eth0: Supported ports: [ TP ] Supported link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full 1000baseT/Full Supports auto-negotiation: Yes Advertised link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full 1000baseT/Full Advertised pause frame use: No Advertised auto-negotiation: Yes Speed: 1000Mb/s Duplex: Full Port: Twisted Pair PHYAD: 1 Transceiver: internal Auto-negotiation: on MDI-X: off Supports Wake-on: pumbg Wake-on: d Current message level: 0x00000001 (1) Link detected: yes nfs01b [root@nfs01b grid]# ifconfig -a bond0 Link encap:Ethernet HWaddr 00:1D:09:71:97:3F inet addr:10.198.1.111 Bcast:10.198.1.255 Mask:255.255.255.0 inet6 addr: fe80::21d:9ff:fe71:973f/64 Scope:Link UP BROADCAST RUNNING MASTER MULTICAST MTU:9000 Metric:1 RX packets:1408007135 errors:603 dropped:0 overruns:0 frame:603 TX packets:1555843721 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:1339715982935 (1.2 TiB) TX bytes:8184089223283 (7.4 TiB) bond0:1 Link encap:Ethernet HWaddr 00:1D:09:71:97:3F inet addr:10.198.1.113 Bcast:10.198.1.255 Mask:255.255.255.0 UP BROADCAST RUNNING MASTER MULTICAST MTU:9000 Metric:1 bond0:3 Link encap:Ethernet HWaddr 00:1D:09:71:97:3F inet addr:10.198.1.115 Bcast:10.198.1.255 Mask:255.255.255.0 UP BROADCAST RUNNING MASTER MULTICAST MTU:9000 Metric:1 eth0 Link encap:Ethernet HWaddr 00:1D:09:71:97:3F UP BROADCAST RUNNING SLAVE MULTICAST MTU:9000 Metric:1 RX packets:399947840 errors:0 dropped:0 overruns:0 frame:0 TX packets:7672 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:2000 RX bytes:300127044161 (279.5 GiB) TX bytes:980800 (957.8 KiB) Interrupt:16 Memory:f8000000-f8012800 eth1 Link encap:Ethernet HWaddr 00:1D:09:71:97:3F UP BROADCAST RUNNING SLAVE MULTICAST MTU:9000 Metric:1 RX packets:274332454 errors:603 dropped:0 overruns:0 frame:603 TX packets:11798913 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:2000 RX bytes:191624805026 (178.4 GiB) TX bytes:31845712465 (29.6 GiB) Interrupt:16 Memory:f4000000-f4012800 lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:44 errors:0 dropped:0 overruns:0 frame:0 TX packets:44 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:3454 (3.3 KiB) TX bytes:3454 (3.3 KiB) p1p1 Link encap:Ethernet HWaddr 00:15:17:6C:E4:88 inet addr:10.1.0.151 Bcast:10.1.0.255 Mask:255.255.255.0 inet6 addr: fe80::215:17ff:fe6c:e488/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:9000 Metric:1 RX packets:514583964 errors:0 dropped:0 overruns:0 frame:0 TX packets:328238028 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:2000 RX bytes:3551414852969 (3.2 TiB) TX bytes:402439759172 (374.8 GiB) Interrupt:16 Memory:fc5e0000-fc600000 p1p2 Link encap:Ethernet HWaddr 00:1D:09:71:97:3F UP BROADCAST RUNNING SLAVE MULTICAST MTU:9000 Metric:1 RX packets:733726841 errors:0 dropped:0 overruns:0 frame:0 TX packets:1544037136 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:2000 RX bytes:847964133748 (789.7 GiB) TX bytes:8152242530018 (7.4 TiB) Interrupt:17 Memory:fc5a0000-fc5c0000 p2p1 Link encap:Ethernet HWaddr 00:15:17:6C:E5:AE inet addr:10.1.0.153 Bcast:10.1.0.255 Mask:255.255.255.0 inet6 addr: fe80::215:17ff:fe6c:e5ae/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:9000 Metric:1 RX packets:545714107 errors:0 dropped:0 overruns:0 frame:0 TX packets:345694600 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:2000 RX bytes:3799060983542 (3.4 TiB) TX bytes:411813015556 (383.5 GiB) Interrupt:16 Memory:fc3e0000-fc400000 p2p2 Link encap:Ethernet HWaddr 00:15:17:6C:E5:AF inet addr:10.1.0.155 Bcast:10.1.0.255 Mask:255.255.255.0 inet6 addr: fe80::215:17ff:fe6c:e5af/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:9000 Metric:1 RX packets:484645571 errors:0 dropped:0 overruns:0 frame:0 TX packets:310892019 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:2000 RX bytes:3346723600136 (3.0 TiB) TX bytes:385654579450 (359.1 GiB) Interrupt:17 Memory:fc3a0000-fc3c0000 [root@nfs01b grid]# cat /proc/net/bonding/bond0 Ethernet Channel Bonding Driver: v3.6.0 (September 26, 2009) Bonding Mode: IEEE 802.3ad Dynamic link aggregation Transmit Hash Policy: layer2 (0) MII Status: up MII Polling Interval (ms): 100 Up Delay (ms): 0 Down Delay (ms): 0 802.3ad info LACP rate: slow Aggregator selection policy (ad_select): stable Active Aggregator Info: Aggregator ID: 1 Number of ports: 3 Actor Key: 17 Partner Key: 10010 Partner Mac Address: 00:1b:ed:80:17:c0 Slave Interface: eth0 MII Status: up Speed: 1000 Mbps Duplex: full Link Failure Count: 3 Permanent HW addr: 00:1d:09:71:97:3f Aggregator ID: 1 Slave queue ID: 0 Slave Interface: eth1 MII Status: up Speed: 1000 Mbps Duplex: full Link Failure Count: 3 Permanent HW addr: 00:1d:09:71:97:41 Aggregator ID: 1 Slave queue ID: 0 Slave Interface: p1p2 MII Status: up Speed: 1000 Mbps Duplex: full Link Failure Count: 2 Permanent HW addr: 00:15:17:6c:e4:89 Aggregator ID: 1 Slave queue ID: 0 [root@nfs01b grid]# mii-tool -v eth0 eth0: negotiated 100baseTx-FD flow-control, link ok product info: vendor 00:08:18, model 54 rev 6 basic mode: autonegotiation enabled basic status: autonegotiation complete, link ok capabilities: 100baseTx-FD 100baseTx-HD 10baseT-FD 10baseT-HD advertising: 100baseTx-FD 100baseTx-HD 10baseT-FD 10baseT-HD flow-control link partner: 100baseTx-FD 100baseTx-HD 10baseT-FD 10baseT-HD flow-control [root@nfs01b grid]# mii-tool -v eth1 eth1: negotiated 100baseTx-FD flow-control, link ok product info: vendor 00:08:18, model 54 rev 6 basic mode: autonegotiation enabled basic status: autonegotiation complete, link ok capabilities: 100baseTx-FD 100baseTx-HD 10baseT-FD 10baseT-HD advertising: 100baseTx-FD 100baseTx-HD 10baseT-FD 10baseT-HD flow-control link partner: 100baseTx-FD 100baseTx-HD 10baseT-FD 10baseT-HD flow-control [root@nfs01b grid]# mii-tool -v p1p2 p1p2: negotiated 100baseTx-FD flow-control, link ok product info: vendor 00:aa:00, model 56 rev 0 basic mode: autonegotiation enabled basic status: autonegotiation complete, link ok capabilities: 100baseTx-FD 100baseTx-HD 10baseT-FD 10baseT-HD advertising: 100baseTx-FD 100baseTx-HD 10baseT-FD 10baseT-HD flow-control link partner: 100baseTx-FD 100baseTx-HD 10baseT-FD 10baseT-HD flow-control [root@nfs01b grid]# cat /etc/exports /data01a dss01a(async,no_root_squash,rw,no_subtree_check) dss01b(async,no_root_squash,rw,no_subtree_check) dss01c(async,no_root_squash,rw,no_subtree_check) /data01b dss01a(async,no_root_squash,rw,no_subtree_check) dss01b(async,no_root_squash,rw,no_subtree_check) dss01c(async,no_root_squash,rw,no_subtree_check) /data01c dss01a(async,no_root_squash,rw,no_subtree_check) dss01b(async,no_root_squash,rw,no_subtree_check) dss01c(async,no_root_squash,rw,no_subtree_check) [root@dss01c ~]# cat /etc/fstab | grep nfs omadvnfs01-nfs-a:/data01a /data01a nfs rw,suid,dev,exec,auto,nouser,async,noatime,nodiratime,_netdev,rsize=32768,wsize=32768,nolock,nfsvers =3 0 0 omadvnfs01-nfs-b:/data01b /data01b nfs rw,suid,dev,exec,auto,nouser,async,noatime,nodiratime,_netdev,rsize=32768,wsize=32768,nolock,nfsvers =3 0 0 omadvnfs01-nfs-c:/data01c /data01c nfs rw,suid,dev,exec,auto,nouser,async,noatime,nodiratime,_netdev,rsize=32768,wsize=32768,nolock,nfsvers =3 0 0 |
Recall that dss01c can communicate to nfs01a just fine. It's just to nfs01b. So one can assume the network config of dss01c and nfs01b independently are good just problematic when talking to each other. Also, it's just via NFS. I can do netcat tests between the hosts and it pans out fine.
|
Could path MTU discovery be different between them - affecting fragmentation?
Are NFS s/w versions and export/mount options the same? Is the workload just NFS write - nothing else such as compilation or search paths or user lookups causing more work to be done? |
Quote:
Quote:
Quote:
|
Hi TD3201,
Did you manage to resolve your issue? I have a similar problem and cannot identify any further stuff to check/correct. http://www.linuxquestions.org/questi...ow-4175442032/ Thanks, - Ned |
All times are GMT -5. The time now is 07:01 PM. |