nfs share can't be unmounted, "device is busy"

pingu · 10-22-2014, 06:57 AM

I'm running oVirt on CentOS-6.5
Problem is the nfs shares used by oVirt can't be unmounted, which leads to the server hanging at reboot.
The server hangs at unmounting with message "umount.nfs: /rhev/data-center/mnt/vnode1.myserver.se:_mnt_exports_data: device is busy".
Same when I try to unmount manually even when no vm's are running.

Code:

lsof /rhev/data-center/mnt/vnode1.myserver.se:_mnt_exports_data
COMMAND    PID    USER   FD   TYPE DEVICE    SIZE/OFF       NODE NAME
sanlock  25044 sanlock    9u   REG   0,28     1048576  536871044 /rhev/data-center/mnt/vnode1.myserver.se:_mnt_exports_data/c63c95f1-3985-41f3-864e-dd6f5dd90c85/dom_md/ids (vnode1.myserver.se:/mnt/exports/data)
sanlock  25044 sanlock   11u   REG   0,28     1048576 1074695842 /rhev/data-center/mnt/vnode1.myserver.se:_mnt_exports_data/c63c95f1-3985-41f3-864e-dd6f5dd90c85/ha_agent/hosted-engine.lockspace (vnode1.myserver.se:/mnt/exports/data)
qemu-kvm 28597    qemu   14u   REG   0,28 26843545600  536871048 /rhev/data-center/mnt/vnode1.myserver.se:_mnt_exports_data/c63c95f1-3985-41f3-864e-dd6f5dd90c85/images/ae55adb1-2f74-4825-a317-3fb193a0f4ae/f8fad855-cc47-4b42-b1ff-bc559eff9859 (vnode1.myserver.se:/mnt/exports/data)

Trying to stop sanlock with "service sanlock stop" fails, no explanation.
There is nothing useful in /var/log/messages or /var/log/sanlock.log
How can I get the server to reboot properly?

vwtech · 10-22-2014, 10:12 AM

What happens if you kill the PID's that have those files open, is that an option that you can attempt?

dubnik · 10-23-2014, 02:13 AM

Hi

try lazy umnount

#umount -l

pingu · 10-23-2014, 12:15 PM

Thanks for your suggestions!
However, both tried no success.
Killing the pid leaves the shares still unmountable, don't remember exactly but "subsys" keeps them locked.
After the kill they are also completely inaccessible.
I tried deleting /var/lock/subsys/sanlock but still no success.

umount -l & umount -f just gives "device is busy", waited about half an hour before rebooting but still hangs.

deep27ak · 11-06-2014, 03:41 AM

Try removing the nfs entry from /etc/mtab

also use the below command

Code:

# lsof +aL1 /mount/point

then

Code:

# kill -9 PID