LinuxQuestions.org
Visit Jeremy's Blog.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Networking
User Name
Password
Linux - Networking This forum is for any issue related to networks or networking.
Routing, network cards, OSI, etc. Anything is fair game.

Notices


Reply
  Search this Thread
Old 12-18-2012, 04:58 PM   #1
evillene
LQ Newbie
 
Registered: Dec 2012
Posts: 10

Rep: Reputation: Disabled
Two NFS Mounts on RHEL - One fast, one super-slow...


Hello,

I have a linux server (lnx3) with 2 NFS mounts pointing to two different servers (fs1 and fs2), one performing well, the other one performing way slower.

They are configured the same way, are on the same network and I see no reason for this slowness.

Can you help find out the reason for that?

Additional info below.

Many thanks,
- Ned

=========================================

Attempts with no success:
* Disabled firewalls
* Played with nfs options (e.g., wsize and rsize)

=========================================

> cat /etc/redhat-release

lnx3: Red Hat Enterprise Linux Server release 5.7
fs1: Red Hat Enterprise Linux Client release 5
fs2: Red Hat Enterprise Linux Server release 5.7


============================================

fstab on lnx3:

[root@lnx3 ~]# cat /etc/fstab
/dev/VolGroup00/LogVol00 / ext3 defaults 1 1
LABEL=/boot /boot ext3 defaults 1 2
tmpfs /dev/shm tmpfs defaults 0 0
devpts /dev/pts devpts gid=5,mode=620 0 0
sysfs /sys sysfs defaults 0 0
proc /proc proc defaults 0 0
/dev/VolGroup00/LogVol01 swap swap defaults 0 0

fs1:/tools /tools nfs defaults 0 0
fs1:/workarea /workarea nfs defaults 0 0
fs2:/home /home nfs defaults 0 0
fs1:/rtk /rtk nfs defaults 0 0

# SMB
\\192.168.0.148\backup1 /mnt/backup1 cifs user=guest,password=xxx,file_mod=0550,uid=root,gid=ipdesign,dir_mode=0550 0 0
\\192.168.0.148\backup2 /mnt/backup2 cifs user=guest,password=xxx,file_mod=0550,uid=root,gid=ipdesign,dir_mode=0550 0 0
\\192.168.0.6\abc /mnt/abc cifs password=xxx,uid=0,file_mod=0770,dir_mode=0770,gid=502,username=labo 0 0
#fs2
fs2:/home/userdata /userdata nfs defaults 0 0


===================================================

[root@fs1 evilleneuve]# /sbin/mii-tool -v
eth0: no link
product info: vendor 00:aa:00, model 56 rev 0
basic mode: autonegotiation enabled
basic status: no link
capabilities: 100baseTx-FD 100baseTx-HD 10baseT-FD 10baseT-HD
advertising: 100baseTx-FD 100baseTx-HD 10baseT-FD 10baseT-HD flow-control
eth1: negotiated 100baseTx-FD, link ok
product info: vendor 00:aa:00, model 56 rev 0
basic mode: autonegotiation enabled
basic status: autonegotiation complete, link ok
capabilities: 100baseTx-FD 100baseTx-HD 10baseT-FD 10baseT-HD
advertising: 100baseTx-FD 100baseTx-HD 10baseT-FD 10baseT-HD flow-control
link partner: 100baseTx-FD 100baseTx-HD 10baseT-FD 10baseT-HD



[root@fs2 ~]# /sbin/mii-tool -v
SIOCGMIIPHY on 'eth0' failed: Resource temporarily unavailable
SIOCGMIIPHY on 'eth1' failed: Resource temporarily unavailable
eth2: negotiated 100baseTx-FD, link ok
product info: vendor 00:55:00, model 59 rev 1
basic mode: autonegotiation enabled
basic status: autonegotiation complete, link ok
capabilities: 100baseTx-FD 100baseTx-HD 10baseT-FD 10baseT-HD
advertising: 100baseTx-FD 100baseTx-HD 10baseT-FD 10baseT-HD flow-control
link partner: 100baseTx-FD 100baseTx-HD 10baseT-FD 10baseT-HD
eth3: negotiated 100baseTx-FD, link ok
product info: vendor 00:55:00, model 59 rev 1
basic mode: autonegotiation enabled
basic status: autonegotiation complete, link ok
capabilities: 100baseTx-FD 100baseTx-HD 10baseT-FD 10baseT-HD
advertising: 100baseTx-FD 100baseTx-HD 10baseT-FD 10baseT-HD flow-control
link partner: 100baseTx-FD 100baseTx-HD 10baseT-FD 10baseT-HD



===================================================

Times monitored on lnx3:

[root@lnx3 ~]# cd /tools/test_mount_fs1/
[root@lnx3 test_mount_fs1]# time tar xjf sysstat-10.0.0.tar.bz2

real 0m0.784s
user 0m0.090s
sys 0m0.012s

[root@lnx3 test_mount_fs1]# cd /home/evilleneuve/test_mount_fs2/
[root@lnx3 test_mount_fs2]# time tar xjf sysstat-10.0.0.tar.bz2

real 0m16.035s
user 0m0.119s
sys 0m0.038s
 
Old 12-19-2012, 03:53 AM   #2
evillene
LQ Newbie
 
Registered: Dec 2012
Posts: 10

Original Poster
Rep: Reputation: Disabled
Additional info:

When copying one large file, fs2 mount is faster than fs1
So, the issue affects specifically the copying of many small files.

[root@lnx4 test_mount_fs2]# time cp /tmp/20120215_104551.mp4 /home/evilleneuve/test_mount_fs2/

real 0m9.952s
user 0m0.000s
sys 0m0.560s
[root@lnx4 test_mount_fs2]# time cp /tmp/20120215_104551.mp4 /tools/test_mount_fs1/

real 0m14.875s
user 0m0.007s
sys 0m0.948s
[root@lnx4 test_mount_fs2]# ll /tmp/20120215_104551.mp4
-rwxr--r-- 1 root root 438624945 Dec 8 16:21 /tmp/20120215_104551.mp4
 
Old 12-20-2012, 02:43 AM   #3
evillene
LQ Newbie
 
Registered: Dec 2012
Posts: 10

Original Poster
Rep: Reputation: Disabled
Additional information:
* I have applied all recommended system updates
* I have removed the port bonding for the server at cause (fs2), to have comparable settings

Despite all of this, I see consistently this performance issue and cannot trace it back to any configuration or other factor.
The numerous threads I have read on this subject have not helped.

Any assistance is much appreciated...
 
Old 12-20-2012, 02:54 AM   #4
evillene
LQ Newbie
 
Registered: Dec 2012
Posts: 10

Original Poster
Rep: Reputation: Disabled
From the ifconfig, I do not see much difference, except that fs1 (fast mount) is configured for IPv6 while fs2 is only for IPv4.

[root@fs1 ~]# ifconfig -a
eth0 Link encap:Ethernet HWaddr 00:14:22:18:4C:C7
BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 lg file transmission:1000
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)
Adresse de base:0xecc0 Mémoire:fe6e0000-fe700000

eth1 Link encap:Ethernet HWaddr 00:14:22:18:4C:C8
inet adr:192.168.0.149 Bcast:192.168.0.255 Masque:255.255.255.0
adr inet6: fe80::214:22ff:fe18:4cc8/64 Scope:Lien
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:1518258266 errors:6 dropped:146563 overruns:0 frame:3
TX packets:1906838427 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 lg file transmission:1000
RX bytes:2029403837 (1.8 GiB) TX bytes:3330457862 (3.1 GiB)
Adresse de base:0xdcc0 Mémoire:fe4e0000-fe500000

lo Link encap:Boucle locale
inet adr:127.0.0.1 Masque:255.0.0.0
adr inet6: ::1/128 Scope:Hôte
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:218264839 errors:0 dropped:0 overruns:0 frame:0
TX packets:218264839 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 lg file transmission:0
RX bytes:488297801 (465.6 MiB) TX bytes:488297801 (465.6 MiB)

sit0 Link encap:IPv6-dans-IPv4
NOARP MTU:1480 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 lg file transmission:0
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)


[root@fs2 ~]# ifconfig -a
eth0 Link encap:Ethernet HWaddr 00:1E:4F:39:20:07
BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)
Interrupt:169 Memory:f8000000-f8012800

eth1 Link encap:Ethernet HWaddr 00:1E:4F:39:20:09
BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)
Interrupt:169 Memory:f4000000-f4012800

eth2 Link encap:Ethernet HWaddr A0:36:9F:0E:05:26
BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)
Memory:fc500000-fc600000

eth3 Link encap:Ethernet HWaddr A0:36:9F:0E:05:27
inet addr:192.168.0.142 Bcast:192.168.0.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:67775325 errors:0 dropped:0 overruns:0 frame:0
TX packets:91717751 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:32434955218 (30.2 GiB) TX bytes:127994773000 (119.2 GiB)
Memory:fc400000-fc500000

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:29203 errors:0 dropped:0 overruns:0 frame:0
TX packets:29203 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:22320608 (21.2 MiB) TX bytes:22320608 (21.2 MiB)
 
Old 12-20-2012, 03:17 AM   #5
evillene
LQ Newbie
 
Registered: Dec 2012
Posts: 10

Original Poster
Rep: Reputation: Disabled
Note: in my initial post, the first two lines of mii-tool say:
SIOCGMIIPHY on 'eth0' failed: Resource temporarily unavailable
SIOCGMIIPHY on 'eth1' failed: Resource temporarily unavailable

(these interfaces are supposed to be inactive)
==> would that possibly indicate a configuration issue?
 
Old 12-20-2012, 03:28 AM   #6
evillene
LQ Newbie
 
Registered: Dec 2012
Posts: 10

Original Poster
Rep: Reputation: Disabled
PMTU configuration is similar and tracepath shows no issue...

Last edited by evillene; 12-20-2012 at 04:00 AM.
 
Old 12-20-2012, 04:41 AM   #7
evillene
LQ Newbie
 
Registered: Dec 2012
Posts: 10

Original Poster
Rep: Reputation: Disabled
I compared the kernel parameters and here are the meaningful differences I see:

FS1 (the fast mount server) has 5 values (set to 0) which FS2 hasn't:
* fs.nfs.fscache_from_error
* fs.nfs.fscache_from_pages
* fs.nfs.fscache_to_error
* fs.nfs.fscache_to_pages
* fs.nfs.fscache_uncache_page

shmall and shmmax are 16 times higher on FS2 (the slow mount server) than on FS1

net.ipv4.route.gc_thresh is 8 times higher on FS2 (the slow mount server) than on FS1

Do you see that any of these parameters should be revised/tested to resolve the performance issue?
 
Old 12-24-2012, 06:26 AM   #8
evillene
LQ Newbie
 
Registered: Dec 2012
Posts: 10

Original Poster
Rep: Reputation: Disabled
Definitely running out of options now:

* Tried changing order and options in /etc/exports and /etc/fstab
* Tried stopping all services on fs2 which were not running on fs1
* Tried switching the switch ports between fs1 and fs2
* Tried changing the ethernet hub used on fs2
* Tried changing the kernel parameters shmmax, shmall and net.ipv4.route.gc_thresh to same values as fs1
* Got same results on a minimal-traffic day (Dec, 24 - no users but myself)
* Aligned the Network Config > DNS options of fs2 with those on fs1

I am still getting the same symptoms.

Any help would be much appreciated here...
 
Old 09-16-2013, 04:22 PM   #9
evillene
LQ Newbie
 
Registered: Dec 2012
Posts: 10

Original Poster
Rep: Reputation: Disabled
Solved with NFS async option

Using async option on NFS mount, i got back to normal times for such operations:

/home *(async,no_subtree_check,no_root_squash,no_all_squash,rw)
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
NetApp NFS to RHEL 6 - mounts but can't write jpeery Linux - Enterprise 2 05-23-2012 01:39 AM
NFS mounts not working in AIX 5.3 getting permission denied from RHEL sudheer_pn Linux - Newbie 2 10-06-2008 02:29 AM
nfs write == super slow; read == super fast - problem? BrianK Linux - Networking 4 08-23-2007 11:59 PM
slow NFS/SCP but fast HTTP/FTP darcon3k Linux - Networking 1 10-13-2005 06:22 AM
fast encrypted NFS but slow regular NFS cazzazullu Linux - Networking 0 10-19-2004 11:08 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Networking

All times are GMT -5. The time now is 05:37 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration