LinuxQuestions.org

LinuxQuestions.org (/questions/)
-   Fedora (http://www.linuxquestions.org/questions/fedora-35/)
-   -   Cannot unmount LV device, keeps saying "device busy" (http://www.linuxquestions.org/questions/fedora-35/cannot-unmount-lv-device-keeps-saying-device-busy-4175439782/)

ciforbg 12-03-2012 04:32 AM

[SOLVED] Cannot unmount LV device, keeps saying "device busy"
 
Hi folks,

I am trying to resize ext4 filesystem, mounted on a LV:

Code:

[root@c4 mapper]# df -h /Backup/
Filesystem                Size  Used Avail Use% Mounted on
/dev/mapper/vg_c4-Backup  99G  62G  32G  67% /Backup

I need to unmount it first, but when I do so it shows it unmounted, but the OS is telling me that the resource is busy:

Code:

[root@c4 mapper]# umount -f /Backup
[root@c4 mapper]# df -h /Backup/
Filesystem              Size  Used Avail Use% Mounted on
/dev/mapper/vg_c4-root  9.9G  8.7G  1.2G  89% /

[root@c4 mapper]# e2fsck /dev/mapper/vg_c4-Backup
e2fsck 1.41.14 (22-Dec-2010)
e2fsck: Device or resource busy while trying to open /dev/mapper/vg_c4-Backup
Filesystem mounted or opened exclusively by another program?

According to fuser these are the processes still using this device:

Code:



[root@c4 mapper]# ls -l /dev/vg_c4/
total 0
lrwxrwxrwx 1 root root 7 Dec  3 10:49 Backup -> ../dm-5

[root@c4 mapper]# fuser -c /dev/dm-5
/dev/dm-5:              1  594  617  1265  1301  1305  1306  1310  1312m  1319  1320  1333  1357  1533  1543  1544  1545  1546  1551  1568  1609  1638  1671  1704  1731  1835  1838  1857  1858  1862m  1879  1880  1881  1946  1954  1968  1969  2030  2036  2053  2163  2177  2180m  2182  2190  2232  2244  2246  2272  2323  2331  2333  2334  2342  2345  2346  2347m  2354  2360  2361  2369  2371  2372  2373  2374  2375  2389  2456  2460  2467  2506  2572  2579  2580  2581  2582  2586  2587  2632  2640  2644  2651  2790  3879  3931  3936  4099  4126  6187  9152  9342  9392  9686  9692  9755c  9869  9870 12357 12411 12827

I didn't open a terminal somewhere in /Backup...
No other programs accessing it, nor do some scripts...
I obviously don't want to kill all those processes
I also don't want to reboot

Can you please advise a solution?

Code:

Fedora release 16 (Verne)
Linux c4.c4.lan 3.6.7-4.fc16.x86_64
  LVM version:    2.02.86(2) (2011-07-08)
  Library version: 1.02.65 (2011-07-08)
  Driver version:  4.23.0

Thanks in advance!

MensaWater 12-04-2012 03:00 PM

You're getting device busy because the Linux native multipather has the device (as evidenced by the dm-* name you're seeing).

We just saw this recently with a USB drive we added to a system. To clear it up we did the following:

The multipathd daemon runs to monitor disk events to determine if a disk should be added to multipathing.
When we tested our server we found it had multipathd running.
My theory as to why we were seeing “device busy” messages when trying to mount the new USB device was because multipathd had captured it and was somehow preventing direct access to the /dev/sd* device.
To test that I ran “multipath –l –v2” which says to list multipathing setup with a verbosity level of 2. That command did show the it had created a new mpathf device comprised of the single path /dev/sdh which was the USB.
By examining /etc/multipath/bindings (an ascii file) one can see the “wwid” (unique ID) numbers associated with given multipath devices.
Ours indicated:
mpathf 241736d6564696120
In the above the second item is the wwid.
Part of the multipath setup is to exclude devices that you don’t want under multipath control (typically because there is only a single path – usually this would be the internal disks but as seen in this case should also be USB attached storage). The configuration file for multipath is /etc/multipath.conf. In that file is a “blacklist” section where one enters the wwids of devices that should be excluded.
I edited the file to include the following line in blacklist section:
wwid 241736d6564696120
After that we found we needed to run “service multipathd restart” so the daemon would reread the configuration file.
Additionally I had found we could temporarily remove the device from multipathing (before the restart of multipathd) by running “multipath -f mpathf”.
Note that stopping multipathd does NOT stop your running multipath devices so others will continue to work. The multipathd is simply a daemon that determines what to do with disks when events such as adding them to the system or removing them from the system occur.

ciforbg 12-06-2012 02:51 AM

Hi MensaWater and thanks for your comments.

However I don't think that this is the case here, because multipather isn't running on my node:

[root@c4 Storage]# multipath –l –v2
Dec 06 10:48:31 | /etc/multipath.conf does not exist, blacklisting all devices.
Dec 06 10:48:31 | A sample multipath.conf file is located at
Dec 06 10:48:31 | /usr/share/doc/device-mapper-multipath-0.4.9/multipath.conf
Dec 06 10:48:31 | You can run /sbin/mpathconf to create or modify /etc/multipath.conf
Dec 06 10:48:31 | DM multipath kernel driver not loaded
[root@c4 Storage]# ps -ef |egrep 'multip|mpath'
[root@c4 Storage]#
[root@c4 Storage]# chkconfig --list |grep -i multi

Note: This output shows SysV services only and does not include native
systemd services. SysV configuration data might be overridden by native
systemd configuration.

[root@c4 Storage]# systemctl |grep multi
multi-user.target loaded active active Multi-User
[root@c4 Storage]#

But anyway, this was useful information.
Thanks a lot!!

MensaWater 12-06-2012 07:42 AM

You're probably right - if multipathd wasn't running then nothing multipath wouldn't have grabbed the disk.

You might want to run lsof /backup to see what is busying out the device. Note that one tricky "busy" that can occur is if someone were to cd into the directory then switch user. Sometimes fuser won't show this process since the effective user changed but usually lsof will.

I am curious however, as to what created this link you mentiond:

Quote:

[root@c4 mapper]# ls -l /dev/vg_c4/
total 0
lrwxrwxrwx 1 root root 7 Dec 3 10:49 Backup -> ../dm-5
I'm used to the dm-* devices being created by multipath. Did you previously run multipath?

Is something else running that created the dm-5 or is the dm-5 itself busy?

Have you verified you don't have any loopback mounts (i.e. Backup or dm-5 is also mounted as something else other than /backup?) Run df -h to be sure you understand where all your mounts are coming from.

Also you might want to verify there is nothing in automounter (/etc/auto*) files that is mounting this rather than fstab.

ciforbg 12-07-2012 04:25 PM

Honestly, I didn't remember if I was messing up with multipathd in the past, but I think no.
I am not sure why the naming is dm-like, but for instance my virtual machines under this node are also with block devices named like dm-*, and I can't say why.

I have software raid0 set up plus LVM. On the virtual machines I only have LVM and the naming is like the virtualization host - dm-*.

Quote:

Is something else running that created the dm-5 or is the dm-5 itself busy?
The device itself was busy showing the same processes as above using it.

Quote:

Have you verified you don't have any loopback mounts (i.e. Backup or dm-5 is also mounted as something else other than /backup?) Run df -h to be sure you understand where all your mounts are coming from.
Verified, no loopback mounts.
And no /etc/auto* files too.

However, I've managed to fix this "locking" with a reboot and now I am unable to proceed reproducing this.

Thanks MensaWater for your help!

MensaWater 12-10-2012 09:39 AM

No problem. Glad you got it resolved.

You might want to go to Thread Tools at the top of the thread and mark it as "Solved". It helps others with similar problems in future find your solution.


All times are GMT -5. The time now is 02:07 AM.