Linux - Networking This forum is for any issue related to networks or networking.
Routing, network cards, OSI, etc. Anything is fair game. |
Notices |
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
Are you new to LinuxQuestions.org? Visit the following links:
Site Howto |
Site FAQ |
Sitemap |
Register Now
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
|
 |
12-18-2012, 04:58 PM
|
#1
|
LQ Newbie
Registered: Dec 2012
Posts: 10
Rep: 
|
Two NFS Mounts on RHEL - One fast, one super-slow...
Hello,
I have a linux server (lnx3) with 2 NFS mounts pointing to two different servers (fs1 and fs2), one performing well, the other one performing way slower.
They are configured the same way, are on the same network and I see no reason for this slowness.
Can you help find out the reason for that?
Additional info below.
Many thanks,
- Ned
=========================================
Attempts with no success:
* Disabled firewalls
* Played with nfs options (e.g., wsize and rsize)
=========================================
> cat /etc/redhat-release
lnx3: Red Hat Enterprise Linux Server release 5.7
fs1: Red Hat Enterprise Linux Client release 5
fs2: Red Hat Enterprise Linux Server release 5.7
============================================
fstab on lnx3:
[root@lnx3 ~]# cat /etc/fstab
/dev/VolGroup00/LogVol00 / ext3 defaults 1 1
LABEL=/boot /boot ext3 defaults 1 2
tmpfs /dev/shm tmpfs defaults 0 0
devpts /dev/pts devpts gid=5,mode=620 0 0
sysfs /sys sysfs defaults 0 0
proc /proc proc defaults 0 0
/dev/VolGroup00/LogVol01 swap swap defaults 0 0
fs1:/tools /tools nfs defaults 0 0
fs1:/workarea /workarea nfs defaults 0 0
fs2:/home /home nfs defaults 0 0
fs1:/rtk /rtk nfs defaults 0 0
# SMB
\\192.168.0.148\backup1 /mnt/backup1 cifs user=guest,password=xxx,file_mod=0550,uid=root,gid=ipdesign,dir_mode=0550 0 0
\\192.168.0.148\backup2 /mnt/backup2 cifs user=guest,password=xxx,file_mod=0550,uid=root,gid=ipdesign,dir_mode=0550 0 0
\\192.168.0.6\abc /mnt/abc cifs password=xxx,uid=0,file_mod=0770,dir_mode=0770,gid=502,username=labo 0 0
#fs2
fs2:/home/userdata /userdata nfs defaults 0 0
===================================================
[root@fs1 evilleneuve]# /sbin/mii-tool -v
eth0: no link
product info: vendor 00:aa:00, model 56 rev 0
basic mode: autonegotiation enabled
basic status: no link
capabilities: 100baseTx-FD 100baseTx-HD 10baseT-FD 10baseT-HD
advertising: 100baseTx-FD 100baseTx-HD 10baseT-FD 10baseT-HD flow-control
eth1: negotiated 100baseTx-FD, link ok
product info: vendor 00:aa:00, model 56 rev 0
basic mode: autonegotiation enabled
basic status: autonegotiation complete, link ok
capabilities: 100baseTx-FD 100baseTx-HD 10baseT-FD 10baseT-HD
advertising: 100baseTx-FD 100baseTx-HD 10baseT-FD 10baseT-HD flow-control
link partner: 100baseTx-FD 100baseTx-HD 10baseT-FD 10baseT-HD
[root@fs2 ~]# /sbin/mii-tool -v
SIOCGMIIPHY on 'eth0' failed: Resource temporarily unavailable
SIOCGMIIPHY on 'eth1' failed: Resource temporarily unavailable
eth2: negotiated 100baseTx-FD, link ok
product info: vendor 00:55:00, model 59 rev 1
basic mode: autonegotiation enabled
basic status: autonegotiation complete, link ok
capabilities: 100baseTx-FD 100baseTx-HD 10baseT-FD 10baseT-HD
advertising: 100baseTx-FD 100baseTx-HD 10baseT-FD 10baseT-HD flow-control
link partner: 100baseTx-FD 100baseTx-HD 10baseT-FD 10baseT-HD
eth3: negotiated 100baseTx-FD, link ok
product info: vendor 00:55:00, model 59 rev 1
basic mode: autonegotiation enabled
basic status: autonegotiation complete, link ok
capabilities: 100baseTx-FD 100baseTx-HD 10baseT-FD 10baseT-HD
advertising: 100baseTx-FD 100baseTx-HD 10baseT-FD 10baseT-HD flow-control
link partner: 100baseTx-FD 100baseTx-HD 10baseT-FD 10baseT-HD
===================================================
Times monitored on lnx3:
[root@lnx3 ~]# cd /tools/test_mount_fs1/
[root@lnx3 test_mount_fs1]# time tar xjf sysstat-10.0.0.tar.bz2
real 0m0.784s
user 0m0.090s
sys 0m0.012s
[root@lnx3 test_mount_fs1]# cd /home/evilleneuve/test_mount_fs2/
[root@lnx3 test_mount_fs2]# time tar xjf sysstat-10.0.0.tar.bz2
real 0m16.035s
user 0m0.119s
sys 0m0.038s
|
|
|
12-19-2012, 03:53 AM
|
#2
|
LQ Newbie
Registered: Dec 2012
Posts: 10
Original Poster
Rep: 
|
Additional info:
When copying one large file, fs2 mount is faster than fs1
So, the issue affects specifically the copying of many small files.
[root@lnx4 test_mount_fs2]# time cp /tmp/20120215_104551.mp4 /home/evilleneuve/test_mount_fs2/
real 0m9.952s
user 0m0.000s
sys 0m0.560s
[root@lnx4 test_mount_fs2]# time cp /tmp/20120215_104551.mp4 /tools/test_mount_fs1/
real 0m14.875s
user 0m0.007s
sys 0m0.948s
[root@lnx4 test_mount_fs2]# ll /tmp/20120215_104551.mp4
-rwxr--r-- 1 root root 438624945 Dec 8 16:21 /tmp/20120215_104551.mp4
|
|
|
12-20-2012, 02:43 AM
|
#3
|
LQ Newbie
Registered: Dec 2012
Posts: 10
Original Poster
Rep: 
|
Additional information:
* I have applied all recommended system updates
* I have removed the port bonding for the server at cause (fs2), to have comparable settings
Despite all of this, I see consistently this performance issue and cannot trace it back to any configuration or other factor.
The numerous threads I have read on this subject have not helped.
Any assistance is much appreciated...
|
|
|
12-20-2012, 02:54 AM
|
#4
|
LQ Newbie
Registered: Dec 2012
Posts: 10
Original Poster
Rep: 
|
From the ifconfig, I do not see much difference, except that fs1 (fast mount) is configured for IPv6 while fs2 is only for IPv4.
[root@fs1 ~]# ifconfig -a
eth0 Link encap:Ethernet HWaddr 00:14:22:18:4C:C7
BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 lg file transmission:1000
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)
Adresse de base:0xecc0 Mémoire:fe6e0000-fe700000
eth1 Link encap:Ethernet HWaddr 00:14:22:18:4C:C8
inet adr:192.168.0.149 Bcast:192.168.0.255 Masque:255.255.255.0
adr inet6: fe80::214:22ff:fe18:4cc8/64 Scope:Lien
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:1518258266 errors:6 dropped:146563 overruns:0 frame:3
TX packets:1906838427 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 lg file transmission:1000
RX bytes:2029403837 (1.8 GiB) TX bytes:3330457862 (3.1 GiB)
Adresse de base:0xdcc0 Mémoire:fe4e0000-fe500000
lo Link encap:Boucle locale
inet adr:127.0.0.1 Masque:255.0.0.0
adr inet6: ::1/128 Scope:Hôte
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:218264839 errors:0 dropped:0 overruns:0 frame:0
TX packets:218264839 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 lg file transmission:0
RX bytes:488297801 (465.6 MiB) TX bytes:488297801 (465.6 MiB)
sit0 Link encap:IPv6-dans-IPv4
NOARP MTU:1480 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 lg file transmission:0
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)
[root@fs2 ~]# ifconfig -a
eth0 Link encap:Ethernet HWaddr 00:1E:4F:39:20:07
BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)
Interrupt:169 Memory:f8000000-f8012800
eth1 Link encap:Ethernet HWaddr 00:1E:4F:39:20:09
BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)
Interrupt:169 Memory:f4000000-f4012800
eth2 Link encap:Ethernet HWaddr A0:36:9F:0E:05:26
BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)
Memory:fc500000-fc600000
eth3 Link encap:Ethernet HWaddr A0:36:9F:0E:05:27
inet addr:192.168.0.142 Bcast:192.168.0.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:67775325 errors:0 dropped:0 overruns:0 frame:0
TX packets:91717751 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:32434955218 (30.2 GiB) TX bytes:127994773000 (119.2 GiB)
Memory:fc400000-fc500000
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:29203 errors:0 dropped:0 overruns:0 frame:0
TX packets:29203 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:22320608 (21.2 MiB) TX bytes:22320608 (21.2 MiB)
|
|
|
12-20-2012, 03:17 AM
|
#5
|
LQ Newbie
Registered: Dec 2012
Posts: 10
Original Poster
Rep: 
|
Note: in my initial post, the first two lines of mii-tool say:
SIOCGMIIPHY on 'eth0' failed: Resource temporarily unavailable
SIOCGMIIPHY on 'eth1' failed: Resource temporarily unavailable
(these interfaces are supposed to be inactive)
==> would that possibly indicate a configuration issue?
|
|
|
12-20-2012, 03:28 AM
|
#6
|
LQ Newbie
Registered: Dec 2012
Posts: 10
Original Poster
Rep: 
|
PMTU configuration is similar and tracepath shows no issue...
Last edited by evillene; 12-20-2012 at 04:00 AM.
|
|
|
12-20-2012, 04:41 AM
|
#7
|
LQ Newbie
Registered: Dec 2012
Posts: 10
Original Poster
Rep: 
|
I compared the kernel parameters and here are the meaningful differences I see:
FS1 (the fast mount server) has 5 values (set to 0) which FS2 hasn't:
* fs.nfs.fscache_from_error
* fs.nfs.fscache_from_pages
* fs.nfs.fscache_to_error
* fs.nfs.fscache_to_pages
* fs.nfs.fscache_uncache_page
shmall and shmmax are 16 times higher on FS2 (the slow mount server) than on FS1
net.ipv4.route.gc_thresh is 8 times higher on FS2 (the slow mount server) than on FS1
Do you see that any of these parameters should be revised/tested to resolve the performance issue?
|
|
|
12-24-2012, 06:26 AM
|
#8
|
LQ Newbie
Registered: Dec 2012
Posts: 10
Original Poster
Rep: 
|
Definitely running out of options now:
* Tried changing order and options in /etc/exports and /etc/fstab
* Tried stopping all services on fs2 which were not running on fs1
* Tried switching the switch ports between fs1 and fs2
* Tried changing the ethernet hub used on fs2
* Tried changing the kernel parameters shmmax, shmall and net.ipv4.route.gc_thresh to same values as fs1
* Got same results on a minimal-traffic day (Dec, 24 - no users but myself)
* Aligned the Network Config > DNS options of fs2 with those on fs1
I am still getting the same symptoms.
Any help would be much appreciated here...
|
|
|
09-16-2013, 04:22 PM
|
#9
|
LQ Newbie
Registered: Dec 2012
Posts: 10
Original Poster
Rep: 
|
Solved with NFS async option
Using async option on NFS mount, i got back to normal times for such operations:
/home *(async,no_subtree_check,no_root_squash,no_all_squash,rw)
|
|
|
All times are GMT -5. The time now is 05:37 PM.
|
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.
|
Latest Threads
LQ News
|
|