[SOLVED] Can't unmount -- oops, can't fsck after unmount

upnort · 05-17-2019, 08:12 PM

Slackware 14.2 64-bit.

In my home network I have systems connecting to a server using SSH and NFS. Through the SSH tunnel the system creates a remote NFS mount point.

Often after unmounting, the mount point created on the server remains. The lsof command confirms the mount point is active.

Running umount -f -l on the server fails to release the mount point. I have been unable to release the mount using /usr/bin/fuser -k -m $dir. The only way I have found to release the mount point is reboot or drop to run level 1. That means something in rc.K or called through that script successfully unmounts the mount point.

What should I try next?

Thanks.

Richard Cranium · 05-17-2019, 09:12 PM

Dropping down to runlevel one should shut down the NFS server itself.

In fact...

Code:

# Shut down the NFS server:
if [ -x /etc/rc.d/rc.nfsd ]; then
  /etc/rc.d/rc.nfsd stop
fi

# Kill any processes (typically gam) that would otherwise prevent
# unmounting NFS volumes:
unset FUSER_DELAY
for dir in $(/bin/mount | grep 'type nfs ' | cut -d ' ' -f 3 ) ; do
  echo "Killing processes holding NFS mount $dir open..."
  # Background this to prevent fuser from also blocking shutdown:
  /usr/bin/fuser -k -m $dir &
  FUSER_DELAY=5
done
# If fuser was run, let it have some delay:
if [ ! -z "$FUSER_DELAY" ]; then
  sleep $FUSER_DELAY
fi

# Unmount any NFS, SMB, or CIFS filesystems:
echo "Unmounting remote filesystems:"
/bin/umount -v -a -l -f -r -t nfs,smbfs,cifs | tr -d ' ' | grep successfully | sed "s/:successfullyunmounted/ has been successfully unmounted./g"

That snippet is in /etc/rc.d/rc.K

EDIT: Looking at it, it's odd that FUSER_DELAY is unset going into the loop, but then set after the first call to fuser -k. No, it isn't odd. fuser doesn't care about that environment variable. Derp.

average_user · 05-18-2019, 06:14 AM

It's possible that there are some processes in uninterruptible sleep state. Check what processes are using your mountpoint using 'lsof <MOUNTPOINT>'. Do it as root user.

However, it doesn't explain why dropping to runlevel 1 causes these problematic mountpoints to unmount.

average_user · 05-18-2019, 06:31 AM

Sorry, that may not be correct. More than once couldn't I kill a process in uninterruptible state even with -9 because it was trying to read some files from unmounted NFS share. But it's a side effect of unmounting these shares before killing all processes that use them. However, it doesn't explain why you can't unmount your share.

Another hint - try running this:

Code:

sudo fuser -vm /media/data

I noticed that lsof that I suggested in my previous post cannot always find all files that use the mountpoint. For example, lsof does not list swapfile but fuser does.

*EDIT*:
It would also be great if you posted all messages you got after running 'umount -f -l'.

upnort · 05-18-2019, 11:41 AM

Well shoot me. I described the problem in a horrible way.

The mount point mounts a partition. The mount point does unmount. Yet lsof shows the partition is open. Because of that I cannot perform an fsck on the partition. I receive the infamous /dev/sdXX is in use message from fsck.

The lsof output looks like this:

Code:

jbd2/sdb5 30005             root  cwd       DIR                8,2     4096          2 /
jbd2/sdb5 30005             root  rtd       DIR                8,2     4096          2 /
jbd2/sdb5 30005             root  txt   unknown                                        /proc/30005/exe

After a few hours the problem disappears.

OK. Try again. Based on the replies here, on a whim I restarted the nfsd service. Boom, fsck no longer complained and lsof showed no open files. I don't know why I did not suspect nfs from the beginning. Perhaps sometimes we just need to converse about a topic for solutions to snap into our heads.

To be certain I repeated the exercise a couple more times. Each time verifying that fsck refused to run. Restart nfsd and fsck is happy.

My solution then is in my script to restart the nfsd service. I'll tag the thread solved.