[SOLVED] Cannot unmount LV device, keeps saying "device busy"
I am trying to resize ext4 filesystem, mounted on a LV:
No other programs accessing it, nor do some scripts...
I obviously don't want to kill all those processes
I also don't want to reboot
Can you please advise a solution?
You're getting device busy because the Linux native multipather has the device (as evidenced by the dm-* name you're seeing).
We just saw this recently with a USB drive we added to a system. To clear it up we did the following:
The multipathd daemon runs to monitor disk events to determine if a disk should be added to multipathing.
When we tested our server we found it had multipathd running.
My theory as to why we were seeing “device busy” messages when trying to mount the new USB device was because multipathd had captured it and was somehow preventing direct access to the /dev/sd* device.
To test that I ran “multipath –l –v2” which says to list multipathing setup with a verbosity level of 2. That command did show the it had created a new mpathf device comprised of the single path /dev/sdh which was the USB.
By examining /etc/multipath/bindings (an ascii file) one can see the “wwid” (unique ID) numbers associated with given multipath devices.
In the above the second item is the wwid.
Part of the multipath setup is to exclude devices that you don’t want under multipath control (typically because there is only a single path – usually this would be the internal disks but as seen in this case should also be USB attached storage). The configuration file for multipath is /etc/multipath.conf. In that file is a “blacklist” section where one enters the wwids of devices that should be excluded.
I edited the file to include the following line in blacklist section:
After that we found we needed to run “service multipathd restart” so the daemon would reread the configuration file.
Additionally I had found we could temporarily remove the device from multipathing (before the restart of multipathd) by running “multipath -f mpathf”.
Note that stopping multipathd does NOT stop your running multipath devices so others will continue to work. The multipathd is simply a daemon that determines what to do with disks when events such as adding them to the system or removing them from the system occur.
Hi MensaWater and thanks for your comments.
However I don't think that this is the case here, because multipather isn't running on my node:
[root@c4 Storage]# multipath –l –v2
Dec 06 10:48:31 | /etc/multipath.conf does not exist, blacklisting all devices.
Dec 06 10:48:31 | A sample multipath.conf file is located at
Dec 06 10:48:31 | /usr/share/doc/device-mapper-multipath-0.4.9/multipath.conf
Dec 06 10:48:31 | You can run /sbin/mpathconf to create or modify /etc/multipath.conf
Dec 06 10:48:31 | DM multipath kernel driver not loaded
[root@c4 Storage]# ps -ef |egrep 'multip|mpath'
[root@c4 Storage]# chkconfig --list |grep -i multi
Note: This output shows SysV services only and does not include native
systemd services. SysV configuration data might be overridden by native
[root@c4 Storage]# systemctl |grep multi
multi-user.target loaded active active Multi-User
But anyway, this was useful information.
Thanks a lot!!
You're probably right - if multipathd wasn't running then nothing multipath wouldn't have grabbed the disk.
You might want to run lsof /backup to see what is busying out the device. Note that one tricky "busy" that can occur is if someone were to cd into the directory then switch user. Sometimes fuser won't show this process since the effective user changed but usually lsof will.
I am curious however, as to what created this link you mentiond:
Is something else running that created the dm-5 or is the dm-5 itself busy?
Have you verified you don't have any loopback mounts (i.e. Backup or dm-5 is also mounted as something else other than /backup?) Run df -h to be sure you understand where all your mounts are coming from.
Also you might want to verify there is nothing in automounter (/etc/auto*) files that is mounting this rather than fstab.
Honestly, I didn't remember if I was messing up with multipathd in the past, but I think no.
I am not sure why the naming is dm-like, but for instance my virtual machines under this node are also with block devices named like dm-*, and I can't say why.
I have software raid0 set up plus LVM. On the virtual machines I only have LVM and the naming is like the virtualization host - dm-*.
And no /etc/auto* files too.
However, I've managed to fix this "locking" with a reboot and now I am unable to proceed reproducing this.
Thanks MensaWater for your help!
No problem. Glad you got it resolved.
You might want to go to Thread Tools at the top of the thread and mark it as "Solved". It helps others with similar problems in future find your solution.
|All times are GMT -5. The time now is 10:13 PM.|