Duplicate /dev/dm- devices for multipathed SAN LUN.

zQUEz · 09-13-2011, 05:57 AM

Trying to understand why I see duplicate /dev/dm- devices and what the logic here is.

Example:
I have 2 HBA's, that can each see 4 disk controllers, totalling 8 paths to a disk.
device-mapper-multipath handles the multipathing and registers it as /dev/mapper/mpath7 and also /dev/dm-9.
When I perform an iostat, I also have a dm-10 that appears to be pointing to the same device because the stats are exactly the same.

According to the documentation for dm-mulitpath, I understand that the Linux Kernel chose not to hide the multiple paths. This is evidenced by the numerous sd devices. But the "dm" devices are a multipath, I just don't understand why there are 2 of them for each SAN LUN and what the logic behind it is?

It makes it difficult to script any type of automated performance monitor because I can't guarantee those dm's will remain the same across reboots. Is there some automated way I can trace back the /dev/mapper/mpath* to both /dev/dm-* devices?

AlucardZero · 09-13-2011, 08:08 AM

Can you run "multipath -ll" and post the stanzas for dm-9 and dm-10?

zQUEz · 09-13-2011, 08:58 AM

Sorry, I should have mentioned that multipath only shows 4 multipath devices. In this case, there is no entry for dm-10. Only dm-9:

mpath7 (350002ac0010a0e65) dm-9 3PARdata,VV
[size=500G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=8][active]
\_ 3:0:3:1 sdad 65:208 [active][ready]
\_ 2:0:0:1 sdb 8:16 [active][ready]
\_ 2:0:1:1 sdf 8:80 [active][ready]
\_ 2:0:2:1 sdj 8:144 [active][ready]
\_ 2:0:3:1 sdn 8:208 [active][ready]
\_ 3:0:0:1 sdr 65:16 [active][ready]
\_ 3:0:1:1 sdv 65:80 [active][ready]
\_ 3:0:2:1 sdz 65:144 [active][ready]

Also, I notice if I do a `ls -lR` on /dev/mpath there is no listing of dm-10.
If I do a `find /dev -name dm-10`, I don't get any hits.
I only ever "see" a listing of dm-10 in iostat.

just to be clear, I am using dm-9 and dm-10 as an example, but I actually have 4 SAN LUN's and all have duplicate dm devices.
I am using RHEL5.6.

AlucardZero · 09-13-2011, 05:36 PM

Okay.. I don't see the problem yet? There are 8 paths to the disk, all amalgamated under dm-9.

Can you post this iostat output? Use [code] tags

tommylovell · 09-13-2011, 09:39 PM

I agree with AlucardZero.

Quote:

mpath7 (350002ac0010a0e65) dm-9 3PARdata,VV
[size=500G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=8][active]
\_ 3:0:3:1 sdad 65:208 [active][ready]
\_ 2:0:0:1 sdb 8:16 [active][ready]
\_ 2:0:1:1 sdf 8:80 [active][ready]
\_ 2:0:2:1 sdj 8:144 [active][ready]
\_ 2:0:3:1 sdn 8:208 [active][ready]
\_ 3:0:0:1 sdr 65:16 [active][ready]
\_ 3:0:1:1 sdv 65:80 [active][ready]
\_ 3:0:2:1 sdz 65:144 [active][ready]

What this shows is exactly what the 'multipath -ll' output should look like given the description you gave in your original post.

Quote:

Also, I notice if I do a `ls -lR` on /dev/mpath there is no listing of dm-10.
If I do a `find /dev -name dm-10`, I don't get any hits.
I only ever "see" a listing of dm-10 in iostat.

You should find dm-10 directly under /dev, i.e. /dev/dm-10 if it still exists. Certainly you should have a /dev/dm-9 if that shows in your multipath -ll command.

'find' doesn't appear to work properly against the /dev directory on my RHEL5.6 system, but an 'ls -l /dev/dm-*' does.

So, the issue would be the dm devices.

To start with and as you know, the dm devices are not guaranteed to remain the same from boot to boot. (If you add or remove an LVM LV it'll effect the multipath dm device names.)

Assuming you have coded 'user_friendly_names=yes' in /etc/multipath.conf (see 'man multipath.conf'), then the mpath devices are persistent, but they are not in 'iostat'. 'iostat' reports the actual /dev/dm-n devices and not /dev/mapper/mpathx "alias". As far as I know, there is no way around this.

But to be clear, you're doing an 'iostat', and the command output has both dm-9 and dm-10 shown? Curious. Sorry I have no answer to that.

anomie · 09-13-2011, 09:49 PM

Quote:

Originally Posted by zQUEz

It makes it difficult to script any type of automated performance monitor because I can't guarantee those dm's will remain the same across reboots. Is there some automated way I can trace back the /dev/mapper/mpath* to both /dev/dm-* devices?

Do not use the /dev/dm- devices. (They're for "internal use".)

For your scripting, monitoring, et al. you should only be using the /dev/mapper devices.

zQUEz · 09-14-2011, 06:39 AM

I appreciate all the help and time from you. I have opened a case with Red Hat (might as well use that support once this year right?) and will post what occurs.
Just to add some responses to your comments above though, I guess the meat of what I would like to achieve is to work out how best to monitor/graph/report on disk performance stats / per SAN LUN (not per SAN path). As tommylovell mentioned above, there doesn't appear to be a way to have iostat report on the mpath devices.

So how do other people do it? Surely others need to see trends of disk performance from the OS'es perspective?

Also, the below code blocks might clarify what I am seeing on the server:

Code:

[root@xxxxxxxx ~]# multipath -ll |grep dm-
mpath7 (350002ac0010a0e65) dm-9 3PARdata,VV
mpath6 (350002ac001090e65) dm-8 3PARdata,VV
mpath5 (350002ac0010c0e65) dm-7 3PARdata,VV
mpath4 (350002ac0010b0e65) dm-6 3PARdata,VV

Code:

[root@xxxxxx ~]# iostat |grep dm
sdm               0.26        15.77         3.85   18904104    4615720
dm-0              4.44         3.91        33.75    4684034   40447608
dm-1              0.27         2.64         0.85    3160962    1016264
dm-2              7.07         7.14        52.54    8556650   62968176
dm-3              0.50         0.03         4.00      38162    4790368
dm-4              2.06        13.33        10.94   15976194   13111704
dm-5              1.61         6.60         6.28    7907752    7525792
dm-6             81.94      2169.56       557.96 2600006650  668662808
dm-7            231.41      3009.03      1552.21 3606030514 1860174112
dm-8              5.98       126.92        30.85  152104842   36972944
dm-9           1689.60      3213.17      1128.72 3850681474 1352659496
dm-10          1689.60      3213.17      1128.72 3850674498 1352659496
dm-11             5.98       126.92        30.85  152098306   36972944
dm-12           231.41      3009.02      1552.21 3606023498 1860174112
dm-13            81.94      2169.55       557.96 2599999634  668662808

Code:

[root@xxxxxxx ~]# ls -al /dev/dm*
brw-rw---- 1 root root 253, 6 Aug 31 10:34 /dev/dm-6
brw-rw---- 1 root root 253, 7 Aug 31 10:34 /dev/dm-7
brw-rw---- 1 root root 253, 8 Aug 31 10:34 /dev/dm-8
brw-rw---- 1 root root 253, 9 Aug 31 10:34 /dev/dm-9

Another interesting observation, is I have some RHEL4 servers as well, that not only don't have this problem with duplicate dm's being reported in iostat, but also don't map mpath devices through to dm devices for a multipath command e.g.:

Code:

root@zzzzzzz ~ $ multipath -ll
mpath2 (350002ac000ca0e65)
[size=20 GB][features="1 queue_if_no_path"][hwhandler="0"]
\_ round-robin 0 [prio=8][active]
 \_ 0:0:2:6  sdac 65:192 [active][ready]
 \_ 0:0:3:6  sdan 66:112 [active][ready]
 \_ 1:0:0:6  sday 67:32  [active][ready]
 \_ 1:0:1:6  sdbj 67:208 [active][ready]
 \_ 1:0:2:6  sdbu 68:128 [active][ready]
 \_ 1:0:3:6  sdcf 69:48  [active][ready]
 \_ 0:0:0:6  sdg  8:96   [active][ready]
 \_ 0:0:1:6  sdr  65:16  [active][ready]

tommylovell · 09-14-2011, 08:02 AM

Quote:

So how do other people do it? Surely others need to see trends of disk performance from the OS'es perspective?

I don't even pretend to do this type of thing...

But when our performance and capacity guys asked how they could match up dm- devices and mpath devices, I gave them this info so that they could script something.

Here's how you can get the maj:min numbers for all dm- devices (i.e. the content of /sys/block/dm-*/dev),

Code:

# for d in $(ls /sys/block/dm-*/dev); do echo -n "$d  ";cat $d ; done
/sys/block/dm-0/dev  253:0
/sys/block/dm-1/dev  253:1
/sys/block/dm-2/dev  253:2
/sys/block/dm-3/dev  253:3
/sys/block/dm-4/dev  253:4
/sys/block/dm-5/dev  253:5
/sys/block/dm-6/dev  253:6
/sys/block/dm-7/dev  253:7

This may break if the maintainer rearchitects how Device-Mapper keeps track of it's information. Unfortunately, I can't see where this information is available via a command, which would insulate you from changes.

So, the alternative would be just to pick the major/minor number out of an 'ls -l' command,

Code:

# for d in $(ls /dev/dm-*); do ls -l $d; done
brw-rw---- 1 root root 253, 6 Aug 21 02:47 /dev/dm-6
brw-rw---- 1 root root 253, 7 Aug 21 02:47 /dev/dm-7

This gives you just the non-LVM dm- devices.

And, 'dmsetup' can give you major/minor numbers of the mpath devices so you can match them up,

Code:

# dmsetup ls
mpath1  (253, 7)
mpath0  (253, 6)
rootvg-var      (253, 1)
rootvg-midrange (253, 4)
rootvg-home     (253, 3)
rootvg-opt      (253, 2)
rootvg-swap     (253, 5)
rootvg-root     (253, 0)

And as mentioned before and you are aware of, these can change from boot to boot, so you better get this from your system at the same time you are getting your 'iostat' info.

Good luck with Red Hat. If they don't give you a fix, at least you can get an explanation.

AlucardZero · 09-14-2011, 08:47 AM

ha, that's interesting. Sorry, I don't know about the iostat display problem. Make sure that your sysstat package is at the latest from RedHat and note that they are up to 5.7.

I will say that on a RHEL 6.1 system (sysstat-9.0.4-18.el6), iostat does not display this issue. I don't currently have any RHEL 5 systems hooked up to a SAN.

anomie · 09-14-2011, 09:52 AM

Quote:

Originally Posted by zQUEz

So how do other people do it? Surely others need to see trends of disk performance from the OS'es perspective?

I monitor CPU, memory usage, and local filesystems on the RHEL side. I have hundreds of LUNs, and tracking them - e.g. via sar(1) - would require a lot of logging. SAN storage performance is monitored on the SAN storage side.