NFS share hanging

abhisheks77 · 03-29-2016, 03:08 PM

Hi,
It is RHEL 5.7 and there are multiple NFS shares mounted on this box. df, cd, ls, fuser, almost all command are hanging on /auto/vsc/toms/tpv/prod. Finally I have to kill that pid, to come back on prompt. Can somebody give any pointer, what needs to be checked ? From NAS side, it is still shared, I can see it in output of "showmount -e ptnasdc2 | grep -i tlwbd07"

PHP Code:



[root@tlwbd07 ~]# cat /etc/fstab | grep -i tpv 
ctnasdc1:/root_vdm_5/APPS01/tom_pvd01         /auto/vsc/toms/tpv    nfs    defaults 0 0 
ptnasdc2:/root_vdm_1/Apps1/tom_pvd01_prod     /auto/vsc/toms/tpv/prod     nfs    defaults 0 0 
[root@tlwbd07 ~]#

MensaWater · 03-29-2016, 03:29 PM

Do you see these items in /etc/mnttab? That is what shows you what is actually mounted (or thought to be mounted). /etc/fstab shows you defined mounts but doesn't tell you if they're actually mounted.

It may be you had these mounted then made some change (e.g. a reboot of the NAS) or had a network glitch where the server lost connection to the original mounts and they still show up in mnttab but the access hung. We saw this recently on a RHEL5 system where it hung while the server sharing it to it was rebooted and didn't clear the issue on the server where we'd mounted them even though the sharing server was again sharing.

If it is mnttab and you run "umount -f" on the mounts does it unmount? If not does "umount -l" unmount it?

abhisheks77 · 03-29-2016, 03:56 PM

I was able to unmount it with lazy option (umount -l) but not able to mount it back, it is again hanging.
NAS was never rebooted. From same NAS, there are multiple shares exported to more than 100 server. None had issue, except this one.

PHP Code:



[root@tlwbd07 ~]# ls -l /etc/mnttab 
ls: /etc/mnttab: No such file or directory[root@tlwbd07 ~]#  
[root@tlwbd07 ~]# mount | grep -i tpv 
ctnasdc1:/root_vdm_5/APPS01/tom_pvd01 on /auto/vsc/toms/tpv type nfs (rw,addr=10.63.xx.xx) 
ptnasdc2:/root_vdm_1/Apps1/tom_pvd01_prod on /auto/vsc/toms/tpv/prod type nfs (rw,addr=10.53.xx.xx) 
[root@tlwbd07 ~]# 
[root@tlwbd07 ~]# cat /proc/mounts | grep -i tpv 
ctnasdc1:/root_vdm_5/APPS01/tom_pvd01 /auto/vsc/toms/tpv nfs rw,vers=3,rsize=65536,wsize=65536,hard,proto=tcp,timeo=600,retrans=2,sec=sys,addr=ctnasdc1 0 0 
ptnasdc2:/root_vdm_1/Apps1/tom_pvd01_prod /auto/vsc/toms/tpv/prod nfs rw,vers=3,rsize=65536,wsize=65536,hard,proto=tcp,timeo=600,retrans=2,sec=sys,addr=ptnasdc2 0 0 
[root@tlwbd07 ~]# 
[root@tlwbd07 ~]# umount /auto/vsc/toms/tpv/prod 
umount: /auto/vsc/toms/tpv/prod: device is busy 
umount: /auto/vsc/toms/tpv/prod: device is busy 
[root@tlwbd07 ~]# 
[root@tlwbd07 ~]# fuser -ck /auto/vsc/toms/tpv/prod --> It again hangs and I had to do kill -9

MensaWater · 03-29-2016, 04:03 PM

Sorry I should have said /etc/mtab not /etc/mnttab.

Can you ping the NAS from the server having the issue?

Do you have more than one network path to the NAS from the server having the issue? How is your share configured (i.e. by hostname? by IP or IP range?)?

Are you sure your mount attempt is using the network path you granted? That is if your share allows for hostname, tlwbd07, what IP is associated with that and is that in fact the IP the NAS sees making the request for the mount? I've seen issues wherein we had a hostname in DNS with an IP that we had a NAS querying but the actual request to the NAS was coming over a different interface because we had more than one VLAN on both the mounting server and the NAS appliance.

abhisheks77 · 03-29-2016, 04:24 PM

After unmounting, it is no more in /etc/mtab now.
Yes, I can ping NAS server from tlwbd07.
Same share is mounted on one more server as well (that is another node of this server). There also, it is behaving same. It is presented with hostname from NAS side.
tlwbd07 is very old server and its IP was never changed on server as well on DNS side.
I will check with NAS team as well, if they see some errors on their side.