Duplicate /dev/dm- devices for multipathed SAN LUN.
 

Trying to understand why I see duplicate /dev/dm- devices and what the logic here is.

Example:
I have 2 HBA's, that can each see 4 disk controllers, totalling 8 paths to a disk.
device-mapper-multipath handles the multipathing and registers it as /dev/mapper/mpath7 and also /dev/dm-9.
When I perform an iostat, I also have a dm-10 that appears to be pointing to the same device because the stats are exactly the same.

According to the documentation for dm-mulitpath, I understand that the Linux Kernel chose not to hide the multiple paths. This is evidenced by the numerous sd devices. But the "dm" devices are a multipath, I just don't understand why there are 2 of them for each SAN LUN and what the logic behind it is?

It makes it difficult to script any type of automated performance monitor because I can't guarantee those dm's will remain the same across reboots. Is there some automated way I can trace back the /dev/mapper/mpath* to both /dev/dm-* devices?

Can you run "multipath -ll" and post the stanzas for dm-9 and dm-10?

Sorry, I should have mentioned that multipath only shows 4 multipath devices. In this case, there is no entry for dm-10. Only dm-9:

mpath7 (350002ac0010a0e65) dm-9 3PARdata,VV
[size=500G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=8][active]
\_ 3:0:3:1 sdad 65:208 [active][ready]
\_ 2:0:0:1 sdb 8:16 [active][ready]
\_ 2:0:1:1 sdf 8:80 [active][ready]
\_ 2:0:2:1 sdj 8:144 [active][ready]
\_ 2:0:3:1 sdn 8:208 [active][ready]
\_ 3:0:0:1 sdr 65:16 [active][ready]
\_ 3:0:1:1 sdv 65:80 [active][ready]
\_ 3:0:2:1 sdz 65:144 [active][ready]



Also, I notice if I do a `ls -lR` on /dev/mpath there is no listing of dm-10.
If I do a `find /dev -name dm-10`, I don't get any hits.
I only ever "see" a listing of dm-10 in iostat.

just to be clear, I am using dm-9 and dm-10 as an example, but I actually have 4 SAN LUN's and all have duplicate dm devices.
I am using RHEL5.6.

Okay.. I don't see the problem yet? There are 8 paths to the disk, all amalgamated under dm-9.

Can you post this iostat output? Use [code] tags

I agree with AlucardZero.

What this shows is exactly what the 'multipath -ll' output should look like given the description you gave in your original post.

You should find dm-10 directly under /dev, i.e. /dev/dm-10 if it still exists. Certainly you should have a /dev/dm-9 if that shows in your multipath -ll command.

'find' doesn't appear to work properly against the /dev directory on my RHEL5.6 system, but an 'ls -l /dev/dm-*' does.

So, the issue would be the dm devices.

To start with and as you know, the dm devices are not guaranteed to remain the same from boot to boot. (If you add or remove an LVM LV it'll effect the multipath dm device names.)

Assuming you have coded 'user_friendly_names=yes' in /etc/multipath.conf (see 'man multipath.conf'), then the mpath devices are persistent, but they are not in 'iostat'. 'iostat' reports the actual /dev/dm-n devices and not /dev/mapper/mpathx "alias". As far as I know, there is no way around this.

But to be clear, you're doing an 'iostat', and the command output has both dm-9 and dm-10 shown? Curious. Sorry I have no answer to that.

Do not use the /dev/dm- devices. (They're for "internal use".)

For your scripting, monitoring, et al. you should only be using the /dev/mapper devices.

I appreciate all the help and time from you. I have opened a case with Red Hat (might as well use that support once this year right?) and will post what occurs.
Just to add some responses to your comments above though, I guess the meat of what I would like to achieve is to work out how best to monitor/graph/report on disk performance stats / per SAN LUN (not per SAN path). As tommylovell mentioned above, there doesn't appear to be a way to have iostat report on the mpath devices.

So how do other people do it? Surely others need to see trends of disk performance from the OS'es perspective?

Also, the below code blocks might clarify what I am seeing on the server:



Another interesting observation, is I have some RHEL4 servers as well, that not only don't have this problem with duplicate dm's being reported in iostat, but also don't map mpath devices through to dm devices for a multipath command e.g.:

I don't even pretend to do this type of thing...

But when our performance and capacity guys asked how they could match up dm- devices and mpath devices, I gave them this info so that they could script something.

Here's how you can get the maj:min numbers for all dm- devices (i.e. the content of /sys/block/dm-*/dev),
This may break if the maintainer rearchitects how Device-Mapper keeps track of it's information. Unfortunately, I can't see where this information is available via a command, which would insulate you from changes.

So, the alternative would be just to pick the major/minor number out of an 'ls -l' command,
This gives you just the non-LVM dm- devices.

And, 'dmsetup' can give you major/minor numbers of the mpath devices so you can match them up,
And as mentioned before and you are aware of, these can change from boot to boot, so you better get this from your system at the same time you are getting your 'iostat' info.

Good luck with Red Hat. If they don't give you a fix, at least you can get an explanation.

ha, that's interesting. Sorry, I don't know about the iostat display problem. Make sure that your sysstat package is at the latest from RedHat and note that they are up to 5.7.

I will say that on a RHEL 6.1 system (sysstat-9.0.4-18.el6), iostat does not display this issue. I don't currently have any RHEL 5 systems hooked up to a SAN.

I monitor CPU, memory usage, and local filesystems on the RHEL side. I have hundreds of LUNs, and tracking them - e.g. via sar(1) - would require a lot of logging. SAN storage performance is monitored on the SAN storage side.