LinuxQuestions.org
Latest LQ Deal: Linux Power User Bundle
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 03-31-2014, 11:03 AM   #1
srinishrews
LQ Newbie
 
Registered: Mar 2014
Posts: 16

Rep: Reputation: Disabled
Shutdown problem unmounting NFS shares


I some machines that have NFS mounts. I can mount them without any problems. However, while shutting down the system, it will get as far as 'unmounting file systems' and hang.

Behavior is quite consistent and i tried to force/lazy unmount NFS shares but the system still hangs at the stage where it says unmounting NFS file systems.

Does anyone have any ideas as to what I should be checking to find the cause of the problem?

I've seen a lot of threads before creating this thread but didnt find any working solution. I saw that 'soft' mount or autofs is one of the possible solution but we cant use both the options. Is there anyway to mention some kind of a timeout option to get the system rebooted ignoring the NFS mounts ?

Thanks in advance
 
Old 03-31-2014, 05:29 PM   #2
smallpond
Senior Member
 
Registered: Feb 2011
Location: Massachusetts, USA
Distribution: CentOS 6 (pre-systemd)
Posts: 2,614

Rep: Reputation: 703Reputation: 703Reputation: 703Reputation: 703Reputation: 703Reputation: 703Reputation: 703
This is not a bug. The system is telling you it can't umount because the device is in use.
There are one or more processes with a file or directory open on the NFS mount.
For example, a user's CWD will have that directory open.
You need to stop the processes which are using the volume and cd any users off the volume.
DO NOT force the umount unless you have good backups and lots of free time.
 
Old 04-02-2014, 03:55 PM   #3
srinishrews
LQ Newbie
 
Registered: Mar 2014
Posts: 16

Original Poster
Rep: Reputation: Disabled
I dont see any active processes running on that mount point though. It just takes a lot of time to unmount the NFS file system which is strange. Thats why i'm curious to find out if there is anyway to force the unmount. The hosts just hangs at "Unmounting NFS Filesystems" and its in the same state for hours. Finally i have to force reset the host to get it rebooted.
 
Old 04-02-2014, 05:10 PM   #4
JeremyBoden
Member
 
Registered: Nov 2011
Posts: 939

Rep: Reputation: 174Reputation: 174
Quote:
Originally Posted by srinishrews View Post
I some machines that have NFS mounts. I can mount them without any problems. However, while shutting down the system, it will get as far as 'unmounting file systems' and hang.
Do you unmount your NFS mount points before starting the system shutdown?
 
Old 04-03-2014, 06:23 AM   #5
jpollard
Senior Member
 
Registered: Dec 2012
Location: Washington DC area
Distribution: Fedora, CentOS, Slackware
Posts: 4,604

Rep: Reputation: 1241Reputation: 1241Reputation: 1241Reputation: 1241Reputation: 1241Reputation: 1241Reputation: 1241Reputation: 1241Reputation: 1241
Are you using a "systemd" based init?

If so, you are having a timing problem between the "kill all user processes" and the "dismount all filesystems"...

It is even possible it is a failure in systemd caused by trying to dismount the local disk before dismounting NFS.

And you will need a totally different kind of help, most likely from a distribution specific forum.

Last edited by jpollard; 04-03-2014 at 06:26 AM.
 
Old 04-03-2014, 11:27 AM   #6
suicidaleggroll
LQ Guru
 
Registered: Nov 2010
Location: Colorado
Distribution: OpenSUSE, CentOS
Posts: 5,260

Rep: Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947
Quote:
Originally Posted by smallpond View Post
DO NOT force the umount unless you have good backups and lots of free time.
Or if you are using the sync mount option on your NFS mounts.
 
Old 04-03-2014, 12:29 PM   #7
srinishrews
LQ Newbie
 
Registered: Mar 2014
Posts: 16

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by jpollard View Post
Are you using a "systemd" based init?

If so, you are having a timing problem between the "kill all user processes" and the "dismount all filesystems"...

It is even possible it is a failure in systemd caused by trying to dismount the local disk before dismounting NFS.

And you will need a totally different kind of help, most likely from a distribution specific forum.

This is what i get from strace when trying to manually umount the file system

Quote:
connect(3, {sa_family=AF_INET, sin_port=htons(1234), sin_addr=inet_addr("XX.XX.XX.XX")}, 16) = -1 ECONNREFUSED (Connection refused)
 
Old 04-03-2014, 12:57 PM   #8
jpollard
Senior Member
 
Registered: Dec 2012
Location: Washington DC area
Distribution: Fedora, CentOS, Slackware
Posts: 4,604

Rep: Reputation: 1241Reputation: 1241Reputation: 1241Reputation: 1241Reputation: 1241Reputation: 1241Reputation: 1241Reputation: 1241Reputation: 1241
Quote:
Originally Posted by srinishrews View Post
This is what i get from strace when trying to manually umount the file system
That almost looks like it shutdown the network before it dismounted the NFS mount...
Or is this before you shutdown the system?

Last edited by jpollard; 04-03-2014 at 12:58 PM.
 
Old 04-03-2014, 01:00 PM   #9
srinishrews
LQ Newbie
 
Registered: Mar 2014
Posts: 16

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by jpollard View Post
That almost looks like it shutdown the network before it dismounted the NFS mount...
Or is this before you shutdown the system?
This is when i'm trying to unmount the file system manually using "umount /filesystem"
 
Old 04-03-2014, 01:05 PM   #10
jpollard
Senior Member
 
Registered: Dec 2012
Location: Washington DC area
Distribution: Fedora, CentOS, Slackware
Posts: 4,604

Rep: Reputation: 1241Reputation: 1241Reputation: 1241Reputation: 1241Reputation: 1241Reputation: 1241Reputation: 1241Reputation: 1241Reputation: 1241
If you can mount it manually, you should also be able to dismount it manually.

A failure would imply that the mountd service on the server has failed.
 
Old 04-03-2014, 02:46 PM   #11
srinishrews
LQ Newbie
 
Registered: Mar 2014
Posts: 16

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by jpollard View Post
If you can mount it manually, you should also be able to dismount it manually.

A failure would imply that the mountd service on the server has failed.
I dont have any issues with mouting though. Only umount seems to be the problem. Here is the verbose output of the umount.

Quote:
Trying to umount /filesystem
mount: trying XX.XX.XX.XX prog 100005 vers 3 prot tcp port 1234
mount: Unable to connect to XX.XX.XX.XX:1234, errno 111 (Connection refused)
 
Old 04-03-2014, 04:06 PM   #12
jpollard
Senior Member
 
Registered: Dec 2012
Location: Washington DC area
Distribution: Fedora, CentOS, Slackware
Posts: 4,604

Rep: Reputation: 1241Reputation: 1241Reputation: 1241Reputation: 1241Reputation: 1241Reputation: 1241Reputation: 1241Reputation: 1241Reputation: 1241
what does a "rpcinfo -p <yourserver>" list when the dismount fails. It should have two entries like
Code:
    100005    3   udp  20048  mountd
    100005    3   tcp  20048  mountd
You can also try a "umount.nfs <directory> -v" and see if it gives any more information.
 
Old 04-03-2014, 04:25 PM   #13
srinishrews
LQ Newbie
 
Registered: Mar 2014
Posts: 16

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by jpollard View Post
what does a "rpcinfo -p <yourserver>" list when the dismount fails. It should have two entries like
Code:
    100005    3   udp  20048  mountd
    100005    3   tcp  20048  mountd
You can also try a "umount.nfs <directory> -v" and see if it gives any more information.

Thanks for all the help !!!

Here is the rpcinfo -p <nfsserver> output. my earlier post above already has the output of umount -v

Quote:
program vers proto port
824395111 1 tcp 59979
824395111 1 udp 49187
100011 1 tcp 59978 rquotad
100011 1 udp 49184 rquotad
536870914 1 udp 4658
536870914 1 tcp 4658
100021 3 udp 49175 nlockmgr
100021 2 udp 49175 nlockmgr
100021 1 udp 49175 nlockmgr
100021 4 udp 49175 nlockmgr
100021 3 tcp 59977 nlockmgr
100021 2 tcp 59977 nlockmgr
100021 1 tcp 59977 nlockmgr
100021 4 tcp 59977 nlockmgr
100024 1 udp 59976 status
100024 1 tcp 59976 status
100003 2 udp 2049 nfs
100003 3 udp 2049 nfs
100003 2 tcp 2049 nfs
100003 3 tcp 2049 nfs
140391 1 udp 31491
100005 3 tcp 1234 mountd
100005 2 tcp 1234 mountd
100005 1 tcp 1234 mountd
100005 3 udp 1234 mountd
100005 2 udp 1234 mountd
100005 1 udp 1234 mountd
102660 1 udp 49154
102660 1 tcp 59975
100000 2 udp 111 portmapper
100000 2 tcp 111 portmapper
 
Old 04-03-2014, 04:42 PM   #14
jpollard
Senior Member
 
Registered: Dec 2012
Location: Washington DC area
Distribution: Fedora, CentOS, Slackware
Posts: 4,604

Rep: Reputation: 1241Reputation: 1241Reputation: 1241Reputation: 1241Reputation: 1241Reputation: 1241Reputation: 1241Reputation: 1241Reputation: 1241
could there possibly be an accidental firewall blocking udp on port 1234? (I think UDP is used for the dismount as there is no need for further data, but I'm not sure without digging into the sources).

I usually see these services on port 20048 instead of 1234. (1234 is supposed to be set aside for the Infoseek search agent, and 20048 is set aside for mountd)
 
Old 04-04-2014, 11:36 AM   #15
srinishrews
LQ Newbie
 
Registered: Mar 2014
Posts: 16

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by jpollard View Post
could there possibly be an accidental firewall blocking udp on port 1234? (I think UDP is used for the dismount as there is no need for further data, but I'm not sure without digging into the sources).

I usually see these services on port 20048 instead of 1234. (1234 is supposed to be set aside for the Infoseek search agent, and 20048 is set aside for mountd)

Thanks jpollard.

Not sure why 1234 is used but i compared other systems and that looks like the same across the environment. I can mount the file system successfully though. So mount uses a tcp vs umount uses udp ? I think i forgot to mention that the problems exists only on servers under firewall. the private ones are just fine.

Quote:
[root@XXXXXX ~]# mount -vvvv /file/system
mount: trying XX.XX.XX.XX prog 100003 vers 3 prot tcp port 2049
mount: trying XX.XX.XX.XX prog 100005 vers 3 prot udp port 1234
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Wireless network disconnects before unmounting network shares-shutdown problem grahamw Linux - Networking 2 05-10-2011 03:05 AM
auto unmounting/unmapping of samba shares Avinazz Linux - Security 1 09-14-2010 10:54 AM
unmounting all nfs filesystems without unmounting local ones kaplan71 Linux - General 1 03-05-2009 12:00 PM
NFS - Shares (File Caching Problem) smaida Linux - Networking 4 04-24-2004 01:39 AM
seeing nfs shares problem (firewall) rosscopeeko Linux - General 3 03-17-2004 04:46 PM


All times are GMT -5. The time now is 03:39 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration