LinuxQuestions.org - mkfs cannot open <device>: Device or resource busy

- Fedora (https://www.linuxquestions.org/questions/fedora-35/)

- - mkfs cannot open <device>: Device or resource busy (https://www.linuxquestions.org/questions/fedora-35/mkfs-cannot-open-device-device-or-resource-busy-4175484362/)

mkfs cannot open <device>: Device or resource busy

I have a 2TB disk that I am trying to configure as a single partition.

Code:

# fdisk -l /dev/sdb

Disk /dev/sdb: 2000.4 GB, 2000398934016 bytes, 3907029168 sectors

Units = sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 4096 bytes

I/O size (minimum/optimal): 4096 bytes / 4096 bytes

Disk label type: dos

Disk identifier: 0x498d2e4b



  Device Boot      Start        End      Blocks  Id  System

/dev/sdb1            2048  3907029167  1953513560  83  Linux

I've created the partition and can see the device files exist.

Code:

brw-rw---- 1 root disk 8, 16 Nov 12 20:28 /dev/sdb

brw-rw---- 1 root disk 8, 17 Nov 12 20:28 /dev/sdb1



# partprobe -s /dev/sdb

/dev/sdb: msdos partitions 1

When I attempt to format it I get this error:

Code:

mkfs.xfs -i size=512 /dev/sdb1

mkfs.xfs: cannot open /dev/sdb1: Device or resource busy

I get a similar error with different FS types:

Code:

mkfs.ext4 /dev/sdb1

mke2fs 1.42.7 (21-Jan-2013)

/dev/sdb1 is apparently in use by the system; will not make a filesystem here!

It is not mounted or otherwise in use:

Code:

# df -h

Filesystem                      Size  Used Avail Use% Mounted on

/dev/mapper/fedora_unknown-root  50G  3.2G  44G  7% /

devtmpfs                        7.9G    0  7.9G  0% /dev

tmpfs                            7.9G    0  7.9G  0% /dev/shm

tmpfs                            7.9G  444K  7.9G  1% /run

tmpfs                            7.9G    0  7.9G  0% /sys/fs/cgroup

tmpfs                            7.9G  36K  7.9G  1% /tmp

/dev/sda1                        477M  102M  347M  23% /boot

/dev/mapper/fedora_unknown-home  401G  4.3G  377G  2% /home

I checked to see what could be using this partition:

Code:

# lsof /dev/sdb1

# lsof /dev/sdb

COMMAND  PID USER  FD  TYPE DEVICE SIZE/OFF NODE NAME

multipath 375 root  10r  BLK  8,16      0t0 7174 /dev/sdb

So the partition in question is not in use, but the device file for the whole disk is. I thought I was onto something, so I stopped multipathd, checked with lsof and it showed no usage, but it made no difference. I got the same mkfs error.

I've googled around and have checked for md usage, but sdb1 is not listed:

Code:

# cat /proc/mdstat

Personalities :

unused devices: <none>

I'm a bit stuck, any ideas?

I'm running:

Code:

3.11.6-201.fc19.x86_64

Thanks.

Would be really strange since df doesnt show it but have you cat'ed your mtab file to see if for some reason it has an entry on that file for it still? That would cause mfks to believe it is in use. Also does multipath believe that it is using at all either? You say you tried an lsof did you also try an fuser on both the device and the partition? Is there a defunct process that still has a fd open on that drive that lsof isnt seeing?

cat /etc/mtab
multipath -v2
fuser /dev/sdb
fuser /dev/sdb1

Cheers for the reply. For the sake of brevity I hadn't included the multipath output in my original post, so apologies - there are 3 x 2TB drives in the system, in addition to a 500GB drive used for the boot disk:

Code:

# multipath -v2

Nov 13 19:42:38 | multipath.conf +5, invalid keyword: getuid_callout

Nov 13 19:42:38 | multipath.conf +18, invalid keyword: getuid_callout

Nov 13 19:42:38 | ST500DM002-1BD142_Z3TDYHYC: ignoring map

create: WDC_WD20EFRX-68AX9N0_WD-WCC300196274 undef ATA    ,WDC WD20EFRX-68A

size=1.8T features='0' hwhandler='0' wp=undef

`-+- policy='service-time 0' prio=1 status=undef

  `- 1:0:0:0 sdb 8:16 undef ready running

create: WDC_WD20EFRX-68AX9N0_WD-WCC300195339 undef ATA    ,WDC WD20EFRX-68A

size=1.8T features='0' hwhandler='0' wp=undef

`-+- policy='service-time 0' prio=1 status=undef

  `- 2:0:0:0 sdc 8:32 undef ready running

create: WDC_WD20EFRX-68AX9N0_WD-WCC300786906 undef ATA    ,WDC WD20EFRX-68A

size=1.8T features='0' hwhandler='0' wp=undef

`-+- policy='service-time 0' prio=1 status=undef

  `- 3:0:0:0 sdd 8:48 undef ready running

# multipath -F

Nov 13 19:42:44 | multipath.conf +5, invalid keyword: getuid_callout

Nov 13 19:42:44 | multipath.conf +18, invalid keyword: getuid_callout

fuser doesn't show anything else up either:

Code:

# fuser /dev/sdb

# fuser /dev/sdb1

#

Mount tab doesn't show an entry, except for the boot partition:

Code:

# cat /etc/mtab|grep sdb

# cat /etc/mtab|grep sda

/dev/sda1 /boot ext4 rw,relatime,stripe=4,data=ordered 0 0

The only other partition on the system is used by LVM for the root vg:

Code:

# pvs

  PV        VG            Fmt  Attr PSize  PFree

  /dev/sda2  fedora_unknown lvm2 a--  465.27g    0

I have tried creating a small 10GB partition, but still get the same issue with mkfs. I've also created 4 primary partitions on the disk and tried each one, with the same result.

I have noted that when the system boots there isn't a device file for the partition, i.e. /dev/sdb1 doesn't exist. Once I run partprobe this is created.

I can't ever recall having this much hassle in the past - any ideas much appreciated.

Ok, so I found the issue - it is the multipath daemon!! :o
I had stopped and disabled this with systemctl, but it keeps bursting back into life (which I hadn't noticed at first); same after a reboot. It still lists itself as disabled, but starts up anyway?! :scratch:
I was fooled by lsof no longer showing any usage of the device after I stopped multipath.service, but clearly this was not accurate :scratch:

Code:

# systemctl status multipathd.service

multipathd.service - Device-Mapper Multipath Device Controller

  Loaded: loaded (/usr/lib/systemd/system/multipathd.service; disabled)

  Active: active (running) since Thu 2013-11-14 19:18:10 GMT; 5min ago

  Process: 369 ExecStart=/sbin/multipathd (code=exited, status=0/SUCCESS)

  Process: 333 ExecStartPre=/sbin/modprobe dm-multipath (code=exited, status=0/SUCCESS)

 Main PID: 371 (multipathd)

  CGroup: name=systemd:/system/multipathd.service

          ââ371 /sbin/multipathd

I checked the multipath output and it listed my three drives, but not my root disk - not sure why sda was not listed? :scratch:

Code:

# multipath -ll

Nov 14 19:08:55 | multipath.conf +5, invalid keyword: getuid_callout

Nov 14 19:08:55 | multipath.conf +18, invalid keyword: getuid_callout

WDC_WD20EFRX-68AX9N0_WD-WCC300196274 dm-2 ATA    ,WDC WD20EFRX-68A

size=1.8T features='0' hwhandler='0' wp=rw

`-+- policy='service-time 0' prio=1 status=active

  `- 1:0:0:0 sdb 8:16 active ready running

WDC_WD20EFRX-68AX9N0_WD-WCC300786906 dm-4 ATA    ,WDC WD20EFRX-68A

size=1.8T features='0' hwhandler='0' wp=rw

`-+- policy='service-time 0' prio=1 status=active

  `- 3:0:0:0 sdd 8:48 active ready running

WDC_WD20EFRX-68AX9N0_WD-WCC300195339 dm-3 ATA    ,WDC WD20EFRX-68A

size=1.8T features='0' hwhandler='0' wp=rw

`-+- policy='service-time 0' prio=1 status=active

  `- 2:0:0:0 sdc 8:32 active ready running

So I checked the multipath and multipathd man pages and I used multipathd -k, followed by show config this displayed details of the default blacklist. I copied a line from this and added it to /etc/multipath.conf and just for good measure rebooted.

Code:

blacklist {

        devnode "^sd[a-z]"

}

After the reboot no devices are listed under multipathd and I can format a partition! :D

Code:

# mkfs.ext3 /dev/sdb1

mke2fs 1.42.7 (21-Jan-2013)

Filesystem label=

OS type: Linux

Block size=4096 (log=2)

Fragment size=4096 (log=2)

Stride=0 blocks, Stripe width=0 blocks

3276800 inodes, 13107200 blocks

655360 blocks (5.00%) reserved for the super user

First data block=0

Maximum filesystem blocks=4294967296

400 block groups

32768 blocks per group, 32768 fragments per group

8192 inodes per group

Superblock backups stored on blocks:

        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,

        4096000, 7962624, 11239424



Allocating group tables: done

Writing inode tables: done

Creating journal (32768 blocks): done

Writing superblocks and filesystem accounting information: done

I'm new to systemctl and multipathd on Linux, so I must be missing something fundamental here. Can anyone share a bit of knowledge/insight into what is going on here and what end configuration I should aim for?

Cheers.