Hi,
I'm using a server with two FC controllers. Multipath has automatically configured two aliases in /etc/multipath/bindings:
The first alies refers to a dev mapper (/dev/mapper/mpatha - which links to dm-5) and I can find its WWID searching in /dev/disk/by-id.

The problem is that I cannot find the second wwid (which would be mpathb). I don't understand how multipathd came up with that. Where would I find the second wwid? Moreover, given that I'm using 4 FC paths through 2 FC controllers (they show up in multipath -ll), shouldn't mpathb also show up somewhere mapped by dm or something to that effect?

What does "multipath -l -v2" show? It should give you the multipath name, the WWID and any component (/dev/sd*) disks that are part of that multipath. You don't usually need to check /dev/disk/by-id.

You say two paths but typically multipath would not show different WWIDs for the same disk presented via 2 separate paths. It seems to think you have 2 completely separate disks - not 2 paths to the same disk.

Note that if you change a LUN on your array (say from RAID5 to RAID6) its WWID changes. Did you by any chanced do that so that multipath saw first one LUN (e.g. RAID5) then another (e.g. RAID6)? You can of course have (and usually do) multiple LUNs from the same array and each would get its own WWID.

mpatha (360002ac0000000000000002900014f1f) dm-5 3PARdata,VV
size=200G features='2 queue_if_no_path retain_attached_hw_handler' hwhandler='1 alua' wp=rw
`-+- policy='round-robin 0' prio=0 status=active
|- 1:0:0:0 sdb 8:16 active undef running
|- 2:0:0:0 sdd 8:48 active undef running
|- 1:0:3:0 sdc 8:32 active undef running
`- 2:0:3:0 sde 8:64 active undef running
I actually said 4 paths, not 2. 2 FC controllers (I guess) and 4 paths. That literally means that I've got 4 FC cables connected to my server coming from an FC Switch.

No, I didn't change the LUNs array. It's been stable. Actually I only have access to the server itself.

So you say that the WWID (in my case, the number displayed after "mpatha") is not dependent on the server, but only on the SAN? If I connected that LUN to another server, then the WWID would be the same? That's good to know, I'm still struggling with the basics.

I only have one LUN.

So your guess was that the LUN might have been changed and that's the reason why multipath would create those wwid aliases? I would say that mpatha represents one controller and mpathb the other. Or am I interpreting it all wrong?

So the multipath command didn't give you any output for mpathb? That makes it look more like it found something that has since gone away. It would discover USB (thumb) drives if you attached them unless you blacklist them in multipath.conf so that might be a possibility.

You can run the scsi_id command against each /dev/sd* you have to see if any of them show that WWID.

Yes - if you move the LUN from one server to another it should have the same WWID on the other server it had on the original. We use this often when migrating LVM VGs from one server to another (e.g. when replacing old server with a new one). This is another one of the many reasons I champion LVM. It has facilities for exporting VGs from one system and importing them on another.

By the way - you can end up with 4 paths to a disk if the array itself is presenting 2 paths to each of 2 ports on the server side. Some arrays do this so you'll have redundancy on the array itself in addition to redundant ports at host level. The Pure Flasharray for example has two separate control units each with fiber connections. You have to zone each sever path to a fiber connection on each of the 2 array control units so that if one goes down you still have a path to the disk as both array control units can see all the disks. This means even if you had a system with a single port you'd end up with two paths if you zoned to both array ports. (Of course in this scenario if your single host port went down you'd lose both paths which is why it is better to have 2 ports at host level.)

I did creat the lvm at least twice after I installed multipath-tools, meaning I created the pv, vg, then I deleted the vg (if I remember correctly - anyway, at least the vg, if not also the pv), then I recreated it. That shouldn't be the reason why that happened, should it? It would make sense if the WWID does remain the same.

So what you're saying is that even if I have 4 FC cables connected into the cage (the server is a blade) and all 4 cables were assigned to the blade I'm currently using, the server might actually only see 4 ports/connections, that is 4 virtualized paths on 2 ports? How do I make sure it sees what it should be seeing?

No LVM wouldn't have anything to do with seeing extra multipath entries. Multipath is created before LVM. Moreover LVM (On Linux at least) is smart enough to read the LVM information off any disk it sees so would know that your component disks are all the same LVM PV. The downside to using a component (/dev/sd*) device vs a multipath device as your LVM PV is that when that particular component's path goes away LVM doesn't automatically know to use an alternate component device. (I/O would fail at the time that path went away. You could scan PVs to use an alternate disk to recover but there is no reason to do that within LVM if you're using the multipath device since it would automatically use any path it found.)

No I wasn't saying it wouldn't use 4 separate paths to the same LUN if you have 4 ports on the server. I was saying that it is possible to have more paths based on the zoning to the array and gave the example of one array I know does this.

If you have 4 host ports zoned to the array and it presents one LUN down all 4 of those paths then you will see the same LUN (with same WWID but different /dev/sd* components). If however in that same scenario your array presented the LUN down each of, say 3, fiber paths of its own you'd end up with 12 total paths to the LUN (4 host x 3 array).

Arrays and zoning get tricky. Here we zone 2 array ports to 2 fiber SAN switches. We then zone each of 2 fiber HBA ports on the servers to the same fiber SAN switches. This means we end up with 4 paths (2 host ports x 2 array ports) fo each LUN even though we only have 2 fiber cables on the host itself. Moreover just connecting fiber doesn't give you the paths as most SAN switches rely on zoning so they have to know you want a specific array port visible by a specific host port. Also the array itself often has "masking" which is similar to zoning so you have to tell it also you want a specific LUN visible to a specific host. For small arrays this seems cumbersome but large arrays can have dozens of hosts and hundreds of LUNs. Having all hosts see all LUNs could overload system tables and/or memory and also open up risks that you don't want. (e.g. Why let your Development server have access to your Production server's LUNs?)

Years ago on installing a new server using SCSI to array instead of fiber and array masking the server couldn't install because its initial ram disk was too small to handle all the LUNs on the array. We had to disconnect the SCSI to the array to do the OS install then connect the array and reboot so it would see all the LUNs.

In a separate issue we had a LUN added to a VG on one server but on another server it appeared available so someone tried to create a separate VG there using the LUN. This caused it to corrupt the VG on the original server. On resolving the issue on the original server it corrupted the VG on the new server. Although you can scan PVs before using them to see if they have anything on them already most people wouldn't do that on disks they assumed were "unassigned" on a new server. By masking the disks only to the server you want them on you avoid this kind of headache.

1 members found this post helpful.

Thank you for your insight!
Would it be correct to say that zoning happens at SAN level and masking at LUN level?

SAN = Storage Area Network

A traditional full SAN is comprised of:
1) One or more disk arrays that present LUNs
2) One or more hosts that will use the LUNs
3) One or more switches that connect the array(s) to the host(s) via "zoning".
By "SAN Level" if you mean item 3 the answer is yes.

However, it IS possible to do direct fiber connections from an array to a host without a switch.
Also the connections don't have to be fiber SCSI - there are other topologies such iSCSI and FCOE that rely on regular LAN switches. There are many arrays now that will do either iSCSI or Fiber SCSI. Ideally any networking you use this way would be isolated from other LAN connections.
It is possible using iSCSI to use one server to present LUNs to another server so that you don't have a separate disk array.
It is possible you'd do the iSCSI from one host to another so no zoning is involved at all.

"Masking" on an array usually requires you to do a setup much like "Zoning" on a switch in that in both you say which hosts connect to with arrays (on which fiber ports). The "Masking" however also says which disks are assigned to the specific host. All arrays may not call it "masking" but generally they have a way to limit which hosts can access which LUNs.

And of course the above ignores NAS which is a way to share out filesystems from one host to another using things like NFS on Linux and CIFS on Windows. There are also NAS appliances that are designed just to share out filesystems to hosts.

2 members found this post helpful.

Regarding my initial problem, it seems that the issue is not that simple. If I remove or comment line mpathb 3600508b1001ca273b75d7358da14e8e3 in bindings, it automatically adds that line again after I restart the service. So something is definitely there.
in /var/log/kern I've found the following:
I've searched on the internet and somewhere it said that the most common cause for this error is that multipath is trying to map the same storage twice.

That 253:15 looks like a device Major and Minor number. Such a device would be under /dev heirarchy.

If you run the command below (including quotes shown) what does it output:
ls -lR /dev |grep "253, " |grep " 15 ": t

Did you run scsi_id against your /dev/sd* devices to see what WWID they report?

Generally I don't work with the bindings file at all. I focus on multipath.conf setup and use the multipath commands to see details (like the multipath -l -v2 I had you run). Looking at it just now I see a comment saying NOT to edit the bindings file. Also i notice the bindings file on the server with the most array LUNs presented gives me information about "user friendly names" (mpatha, mpathb, mpathc etc... associated with WWIDs even though in multipath.conf I specify alias names to use instead.
So in /etc/multipath/bindings I see entries for several user friendly named devices like these 2 that I previously had from a Hitachi VSP array:
mpathf 360060e8006d142000000d1420000209b
mpathai 360060e8006d142000000d142000020b9

However neither of those still exists on the Hitachi VSP array because we migrated to a Pure FlashArray then removed all VSP LUNs from the server and the VSP.
Even before that remove in /etc/multipath.conf I had those defined respectively as:
That meant my names would be /dev/mapper/hldev209b and /dev/mapper/hldev20b9 even though bindings suggests mpathi and mpathai respectively.

Similarly for a couple of LUNs from the Pure Flasharray bindings shows:
mpathbd 3624a9370dc09ae3756ee45010001101b
mpathbn 3624a9370dc09ae3756ee45010001109c

But multipath.conf has them defined as:
If I tell the multipath command to show me the device seen in bindings with "multipath -l mpathbd" it shows this is actually the alias I gave it:
Which is exactly the same as it shows me if I ask it to show by the alias assigned with "multipath -l pure1101b"
pure1101b (3624a9370dc09ae3756ee45010001101b) dm-56 PURE,FlashArray
size=1.0T features='0' hwhandler='0' wp=rw
`-+- policy='queue-length 0' prio=0 status=active
|- 11:0:1:14 sdbj 67:208 active undef running
`- 12:0:0:14 sdgo 132:64 active undef running
It is also what it shows me if I specify the WWID with "multipath -l 3624a9370dc09ae3756ee45010001101b"
pure1101b (3624a9370dc09ae3756ee45010001101b) dm-56 PURE,FlashArray
size=1.0T features='0' hwhandler='0' wp=rw
`-+- policy='queue-length 0' prio=0 status=active
|- 11:0:1:14 sdbj 67:208 active undef running
`- 12:0:0:14 sdgo 132:64 active undef running

Until looking at it just now I didn't realize bindings track of the user friendly name as well as the WWID name when I assigned an alias because the standard tools don't show the user friendly names but rather the aliases.

So you could try running "multipath -l mpathb" and "multipath -l 3600508b1001ca273b75d7358da14e8e3" to see if either of those will give you further information.

As an additional note my experience is that the WWID for all LUNs from the same array will start with the same characters and it is the trailing characters to correspond to LUN IDs from the arrays. For the Hitachi above you see they start with: 360060e8006d142000000d14200 but then the last 6 characters specify the LUN from the array (in LDEV ID format as Hitachi called them).
On the other hand the ones from the Pure array both start with 624a9370dc09ae3756ee45 then the last characters are different and these last characters correspond to what the Pure array calls "serial numbers" on the LUNs it created.

It is the fact that your 2 device WWIDs don't start the same that leads me to believe they are not both coming from your 3PAR array.

Hi,
The reason why I tinkered with the bindings file was only to see what was going to happen. I wasn't aiming at actually changing the aliases. I knew that multipath.conf would override that anyway. I just wanted to know what would happen - and that result was significant - it kept recreating the line.

Right, so as expected multipath -l 3600508b1001ca273b75d7358da14e8e3 returns nothing and, of course, neither multipath -l mpathb (which is even more obvious, given that there's no associated dev mapper).

ls -lR /dev |grep "253, " |grep " 15 " returns nothing. Under /sys/dev/block the highest number (as it were) is 253:14.
lsblk shows an identical topology (which is normal) for all /dev/sdb to /dev/sde, except for the device numbers (which are unique for each) in front of the /dev/sda, sdb, sdc, sde themselves:
To be honest, I haven't been able to get hold of the 'scsi_id', so I'm trying to find other ways, such as "/dev/disk/by-id"

I've just found out something... weird.
WWID 3600508b1001ca273b75d7358da14e8e3 (so mpathb) belongs to /dev/sda! Which is the SSD inside the blade server.

Apparently scsi_id has moved. On RHEL6 it was /sbin/scsi_id but on RHEL7 it is /lib/udev/scsi_id. That /lib/udev is not in the PATH variable for root. On RHEL6 it was installed as part of the package named udev. On RHEL7 it is part of the systemd package. If you're running a systemd based system scsi_id is likely in a similar location as RHEL7. If not there you can likely find it with "find / -name scsi_id".

It isn't surprising to me that your bindings has your internal disk. As mentioned previously multipath finds all disks unless blacklisted. Here we use DELL Servers with PERC internal RAID controllers for internal disks and blacklist same since there aren't really multiple paths to the disks from the OS' perspective. (i.e. It sees a "virtual disk" [LUN] as a single path from the controller to the OS so as one "disk" even in RAID1 with 2 disks.)

As noted I don't really pay much attention to the bindings file though my own findings suggest I should probably do some cleanup at least of the old Hitachi VSP devices. Also perhaps for some snapshots we create based on what is said in this post: https://community.hpe.com/t5/General...s/td-p/6176457

However, I'd recommend against removing the entry you see for sda given that it likely reflects a real device. You should be able to confirm that with scsi_id.

scsi_id usage samples can be found at:
http://fibrevillage.com/sysadmin/491...mples-on-rhel7

I'm running Debian 9 (Proxmox 5), so yes, it's systemd-based. I also found it under /lib/dev/scsi_id. scsi_id on /dev/sda to /dev/sde says nothing.

Regarding the error in the kern.log, should I create another thread for it?

I reply somewhat late because I have to think things over, etc.

What was you command line on checking /dev/sda etc...? By default scsi_id blacklists so you have to implicitly tell it to treat the device as whitelisted. Run "man scsi_id" for more details.

Doing reading on your ioctl error I found indications this occurs when multipath tries to access a device that is busy. Also what I found says often the major/minor shown in the message does not actually correspond to a device because the mapping was never created. It specifically mentioned this often occurring for boot devices.

In any event it suggests blacklisting the device in multipath.conf then reloading multipath.

You should be able to blacklist the device by adding its WWID to the blacklist section of multipath.conf:
Note: If your blacklist section is already uncommented and has other entries you'd just want to add the wwid line before the final "}" rather than adding a new section like the one shown above.

Here we typically install both the lsscsi and sg3_utils packages to get extra tools. If you run "lsscsi" it should show your drives (not multipaths). If you can identify a specific controller you could blacklist that. Here we blacklist "PERC" as most of the Dell systems we use have PERC internal RAID cards to the local disks which as I mentioned previously aren't multipath devices.

1 members found this post helpful.