[SOLVED] Current hangs in shutdown on fuser command

vulcan59 · 07-14-2021, 10:03 AM

Quote:

Originally Posted by igadoter

So login. Switch to run level 0. But before list all active processes and mounts. If shutting down system in run level 1 works - then I think problem is somewhere else. Besides what do you do to shutdown system? From inside GUI, session manager, switching run level or directly by shutdown command?

telinit 0 hangs as I would expect. I don't go anywhere near a gui. Simple boot the system into the shell, mount the NFS share and do "shutdown -h now"
Looking back, telinit 1 worked. That runs rc.K.
telinit 0 or "shutdown -h" runs rc.0.
Both rc.K and rc.0 run the "fuser -k -m -M" in the background. The significant difference is that rc.0 has the wait command which hangs and rc.K has no wait command.

igadoter · 07-14-2021, 10:16 AM

So test all your mounts. Try in run level 1 to free mounts to find one responsible. In run level 1 there are too few process to cause shutdown hang. So probably it is unfinished transaction on some file system.

vulcan59 · 07-14-2021, 11:42 AM

Quote:

Originally Posted by igadoter

So test all your mounts. Try in run level 1 to free mounts to find one responsible. In run level 1 there are too few process to cause shutdown hang. So probably it is unfinished transaction on some file system.

First of all, there is no network available in runlevel 1. Any NFS mounts prior to switching to runlevel 1 are not there anymore. Anyway, I booted directly to runlevel 1. No network available so I started it. I did the mount of my NFS filesystem. Instantly I did a "shutdown -h now" and the machine shutdown.

Regarding an unfinished transaction. This is a single user machine. I don't see how that can be the case. I have only one NFS mount.

Also, let's not lose sight of the fact this all works ok on Slackware 14.2 which uses fuser -k in the background in the same way and has the wait conmmand.

If I am understanding all this, running rc.0 from runlevel 3 with an NFS mount hangs at the wait during shutdown. Running rc.0 from runlevel 1 with an NFS mount does not hang. This points to some process in runlevel 3 which is not executing in runlevel 1 causing the hang. Is that correct or an I confused!

bassmadrigal · 07-14-2021, 11:12 PM

Quote:

Originally Posted by vulcan59

If I am understanding all this, running rc.0 from runlevel 3 with an NFS mount hangs at the wait during shutdown. Running rc.0 from runlevel 1 with an NFS mount does not hang. This points to some process in runlevel 3 which is not executing in runlevel 1 causing the hang. Is that correct or an I confused!

Try adding the wait command to rc.K and see if it still hangs. I imagine the only reason that works is because it's missing the wait command. I'd bet the fuser command is still hanging, but since rc.K is missing wait, it's probably just pushing through it and gets you back to your terminal.

I'd be curious to see if jobs shows the fuser command after you're at init 1 or if the killall5 command manages to kill it.

igadoter · 07-15-2021, 05:05 AM

Do kind of bisection on rc.* services in /etc/rc.d. This way you may have a chance to find responsible service. Disable half of them and another half till you find which service may cause that issue.

vulcan59 · 07-15-2021, 07:17 AM

Quote:

Originally Posted by bassmadrigal

Try adding the wait command to rc.K and see if it still hangs. I imagine the only reason that works is because it's missing the wait command. I'd bet the fuser command is still hanging, but since rc.K is missing wait, it's probably just pushing through it and gets you back to your terminal.

I added a wait in rc.K after the fuser command but the system still shut down ok. jobs shows nothing.

bassmadrigal · 07-15-2021, 10:11 AM

Quote:

Originally Posted by vulcan59

I added a wait in rc.K after the fuser command but the system still shut down ok. jobs shows nothing.

That's definitely interesting! That means there is either something causing the issue in rc.0 or something preventing it in rc.K. You could try running the individual commands from one and running fuser after each one (without backgrounding it) each time to see if you can find what.

vulcan59 · 09-15-2021, 11:30 AM

Hi again. Other priorities stopped me looking at this but I am now back and think I have found the problem.

First of all, it doesn't matter whether the fuser command is backgrounded or not, the shutdown still hangs.

The problem is that for fuser to kill a process holding an NFS mount open it needs access to the network. However, in rc.6 we stop D-Bus just before the fuser and this takes down my wireless network access causing fuser to hang.

bassmadrigal · 09-15-2021, 04:23 PM

If you swap the stopping of D-Bus to after the fuser command, do you get a successful shutdown?

vulcan59 · 09-16-2021, 11:53 AM

I have to move the stopping of D-Bus to after the following unmount of remote filesyatems as well then I get a clean shutdown.

So, the relevant part of rc.6 now looks like this.

Code:

# Stop the haveged entropy daemon:
if [ -x /etc/rc.d/rc.haveged ]; then
  /etc/rc.d/rc.haveged stop
fi

# Kill any processes (typically gam) that would otherwise prevent
# unmounting NFS volumes:
unset FUSER_DELAY
for dir in $(/bin/mount | grep -e 'type nfs ' -e 'type nfs4 ' | sed -e 's|.* on ||g' | cut -d ' ' -f 1) ; do
  echo "Killing processes holding NFS mount $dir open..."
  # Background this to prevent fuser from also blocking shutdown:
  /usr/bin/fuser -k -M -m "$dir" &
  FUSER_DELAY=5
done
# If fuser was run, let it have some delay:
if [ ! -z "$FUSER_DELAY" ]; then
  sleep $FUSER_DELAY
fi

# Unmount any NFS, SMB, or CIFS filesystems:
echo "Unmounting remote filesystems:"
/bin/umount -v -a -l -f -r -t nfs,nfs4,smbfs,cifs | tr -d ' ' | grep successfully | sed "s/:successfullyunmounted/ has been successfully unmounted./g"

# Stop D-Bus:
if [ -x /etc/rc.d/rc.messagebus ]; then
  /etc/rc.d/rc.messagebus stop
fi

# Try to shut down pppd:
PS="$(ps ax)"
if echo "$PS" | /bin/grep -q -w pppd ; then
  if [ -x /usr/sbin/ppp-off ]; then
    /usr/sbin/ppp-off
  fi
fi