LinuxQuestions.org
Review your favorite Linux distribution.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Distributions > Fedora
User Name
Password
Fedora This forum is for the discussion of the Fedora Project.

Notices

Reply
 
Search this Thread
Old 12-03-2012, 04:32 AM   #1
ciforbg
LQ Newbie
 
Registered: Sep 2007
Posts: 16

Rep: Reputation: 1
[SOLVED] Cannot unmount LV device, keeps saying "device busy"


Hi folks,

I am trying to resize ext4 filesystem, mounted on a LV:

Code:
[root@c4 mapper]# df -h /Backup/
Filesystem                Size  Used Avail Use% Mounted on
/dev/mapper/vg_c4-Backup   99G   62G   32G  67% /Backup
I need to unmount it first, but when I do so it shows it unmounted, but the OS is telling me that the resource is busy:

Code:
[root@c4 mapper]# umount -f /Backup 
[root@c4 mapper]# df -h /Backup/
Filesystem              Size  Used Avail Use% Mounted on
/dev/mapper/vg_c4-root  9.9G  8.7G  1.2G  89% /

[root@c4 mapper]# e2fsck /dev/mapper/vg_c4-Backup 
e2fsck 1.41.14 (22-Dec-2010)
e2fsck: Device or resource busy while trying to open /dev/mapper/vg_c4-Backup
Filesystem mounted or opened exclusively by another program?
According to fuser these are the processes still using this device:

Code:

[root@c4 mapper]# ls -l /dev/vg_c4/
total 0
lrwxrwxrwx 1 root root 7 Dec  3 10:49 Backup -> ../dm-5

[root@c4 mapper]# fuser -c /dev/dm-5
/dev/dm-5:               1   594   617  1265  1301  1305  1306  1310  1312m  1319  1320  1333  1357  1533  1543  1544  1545  1546  1551  1568  1609  1638  1671  1704  1731  1835  1838  1857  1858  1862m  1879  1880  1881  1946  1954  1968  1969  2030  2036  2053  2163  2177  2180m  2182  2190  2232  2244  2246  2272  2323  2331  2333  2334  2342  2345  2346  2347m  2354  2360  2361  2369  2371  2372  2373  2374  2375  2389  2456  2460  2467  2506  2572  2579  2580  2581  2582  2586  2587  2632  2640  2644  2651  2790  3879  3931  3936  4099  4126  6187  9152  9342  9392  9686  9692  9755c  9869  9870 12357 12411 12827
I didn't open a terminal somewhere in /Backup...
No other programs accessing it, nor do some scripts...
I obviously don't want to kill all those processes
I also don't want to reboot

Can you please advise a solution?

Code:
Fedora release 16 (Verne)
Linux c4.c4.lan 3.6.7-4.fc16.x86_64
  LVM version:     2.02.86(2) (2011-07-08)
  Library version: 1.02.65 (2011-07-08)
  Driver version:  4.23.0
Thanks in advance!

Last edited by ciforbg; 12-11-2012 at 04:35 AM. Reason: solved
 
Old 12-04-2012, 03:00 PM   #2
MensaWater
Guru
 
Registered: May 2005
Location: Atlanta Georgia USA
Distribution: Redhat (RHEL), CentOS, Fedora, Debian, FreeBSD, HP-UX, Solaris, SCO
Posts: 6,006
Blog Entries: 5

Rep: Reputation: 783Reputation: 783Reputation: 783Reputation: 783Reputation: 783Reputation: 783Reputation: 783
You're getting device busy because the Linux native multipather has the device (as evidenced by the dm-* name you're seeing).

We just saw this recently with a USB drive we added to a system. To clear it up we did the following:

The multipathd daemon runs to monitor disk events to determine if a disk should be added to multipathing.
When we tested our server we found it had multipathd running.
My theory as to why we were seeing “device busy” messages when trying to mount the new USB device was because multipathd had captured it and was somehow preventing direct access to the /dev/sd* device.
To test that I ran “multipath –l –v2” which says to list multipathing setup with a verbosity level of 2. That command did show the it had created a new mpathf device comprised of the single path /dev/sdh which was the USB.
By examining /etc/multipath/bindings (an ascii file) one can see the “wwid” (unique ID) numbers associated with given multipath devices.
Ours indicated:
mpathf 241736d6564696120
In the above the second item is the wwid.
Part of the multipath setup is to exclude devices that you don’t want under multipath control (typically because there is only a single path – usually this would be the internal disks but as seen in this case should also be USB attached storage). The configuration file for multipath is /etc/multipath.conf. In that file is a “blacklist” section where one enters the wwids of devices that should be excluded.
I edited the file to include the following line in blacklist section:
wwid 241736d6564696120
After that we found we needed to run “service multipathd restart” so the daemon would reread the configuration file.
Additionally I had found we could temporarily remove the device from multipathing (before the restart of multipathd) by running “multipath -f mpathf”.
Note that stopping multipathd does NOT stop your running multipath devices so others will continue to work. The multipathd is simply a daemon that determines what to do with disks when events such as adding them to the system or removing them from the system occur.
 
Old 12-06-2012, 02:51 AM   #3
ciforbg
LQ Newbie
 
Registered: Sep 2007
Posts: 16

Original Poster
Rep: Reputation: 1
Hi MensaWater and thanks for your comments.

However I don't think that this is the case here, because multipather isn't running on my node:

[root@c4 Storage]# multipath –l –v2
Dec 06 10:48:31 | /etc/multipath.conf does not exist, blacklisting all devices.
Dec 06 10:48:31 | A sample multipath.conf file is located at
Dec 06 10:48:31 | /usr/share/doc/device-mapper-multipath-0.4.9/multipath.conf
Dec 06 10:48:31 | You can run /sbin/mpathconf to create or modify /etc/multipath.conf
Dec 06 10:48:31 | DM multipath kernel driver not loaded
[root@c4 Storage]# ps -ef |egrep 'multip|mpath'
[root@c4 Storage]#
[root@c4 Storage]# chkconfig --list |grep -i multi

Note: This output shows SysV services only and does not include native
systemd services. SysV configuration data might be overridden by native
systemd configuration.

[root@c4 Storage]# systemctl |grep multi
multi-user.target loaded active active Multi-User
[root@c4 Storage]#

But anyway, this was useful information.
Thanks a lot!!
 
Old 12-06-2012, 07:42 AM   #4
MensaWater
Guru
 
Registered: May 2005
Location: Atlanta Georgia USA
Distribution: Redhat (RHEL), CentOS, Fedora, Debian, FreeBSD, HP-UX, Solaris, SCO
Posts: 6,006
Blog Entries: 5

Rep: Reputation: 783Reputation: 783Reputation: 783Reputation: 783Reputation: 783Reputation: 783Reputation: 783
You're probably right - if multipathd wasn't running then nothing multipath wouldn't have grabbed the disk.

You might want to run lsof /backup to see what is busying out the device. Note that one tricky "busy" that can occur is if someone were to cd into the directory then switch user. Sometimes fuser won't show this process since the effective user changed but usually lsof will.

I am curious however, as to what created this link you mentiond:

Quote:
[root@c4 mapper]# ls -l /dev/vg_c4/
total 0
lrwxrwxrwx 1 root root 7 Dec 3 10:49 Backup -> ../dm-5
I'm used to the dm-* devices being created by multipath. Did you previously run multipath?

Is something else running that created the dm-5 or is the dm-5 itself busy?

Have you verified you don't have any loopback mounts (i.e. Backup or dm-5 is also mounted as something else other than /backup?) Run df -h to be sure you understand where all your mounts are coming from.

Also you might want to verify there is nothing in automounter (/etc/auto*) files that is mounting this rather than fstab.
 
Old 12-07-2012, 04:25 PM   #5
ciforbg
LQ Newbie
 
Registered: Sep 2007
Posts: 16

Original Poster
Rep: Reputation: 1
Honestly, I didn't remember if I was messing up with multipathd in the past, but I think no.
I am not sure why the naming is dm-like, but for instance my virtual machines under this node are also with block devices named like dm-*, and I can't say why.

I have software raid0 set up plus LVM. On the virtual machines I only have LVM and the naming is like the virtualization host - dm-*.

Quote:
Is something else running that created the dm-5 or is the dm-5 itself busy?
The device itself was busy showing the same processes as above using it.

Quote:
Have you verified you don't have any loopback mounts (i.e. Backup or dm-5 is also mounted as something else other than /backup?) Run df -h to be sure you understand where all your mounts are coming from.
Verified, no loopback mounts.
And no /etc/auto* files too.

However, I've managed to fix this "locking" with a reboot and now I am unable to proceed reproducing this.

Thanks MensaWater for your help!
 
1 members found this post helpful.
Old 12-10-2012, 09:39 AM   #6
MensaWater
Guru
 
Registered: May 2005
Location: Atlanta Georgia USA
Distribution: Redhat (RHEL), CentOS, Fedora, Debian, FreeBSD, HP-UX, Solaris, SCO
Posts: 6,006
Blog Entries: 5

Rep: Reputation: 783Reputation: 783Reputation: 783Reputation: 783Reputation: 783Reputation: 783Reputation: 783
No problem. Glad you got it resolved.

You might want to go to Thread Tools at the top of the thread and mark it as "Solved". It helps others with similar problems in future find your solution.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Disk Device sda3 is 83.57% , "disk device busy" messages - where to look? raghavendra.eranki Linux - General 1 05-11-2012 01:57 PM
Auto-unmount device in kde after performing "device action" newbie to linux Linux - Software 1 01-25-2012 06:28 AM
FC multipath says "device busy " when connecting again. SLES 11 - netapp - emulex permalac Linux - Server 1 01-13-2012 04:45 PM
Force Check done ever time. Unmount won't work on "Busy device"... R4ndom Linux - Newbie 1 04-21-2009 12:52 PM
nvidia insmod: "Device or resource busy" w/GeForce 4 MX KeithSpook Linux - Hardware 4 03-03-2004 12:21 PM


All times are GMT -5. The time now is 05:17 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration