LinuxQuestions.org
Review your favorite Linux distribution.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Server
User Name
Password
Linux - Server This forum is for the discussion of Linux Software used in a server related context.

Notices


Reply
  Search this Thread
Old 01-19-2018, 09:28 AM   #16
vincix
Senior Member
 
Registered: Feb 2011
Distribution: Ubuntu, Centos
Posts: 1,158

Original Poster
Rep: Reputation: 87

Do you have any ideas how multipath would come up with that major/minor? Might it simply have no relationship whatsoever to a certain device and could it be a case of just incrementing the number (given that the last device was 253:14 and in kern.log it finds 253:15)?

I added the following to multipath.conf:
Code:
blacklist {
        wwid "3600508b1001ca273b75d7358da14e8e3"
        }
I also have:
Code:
defaults {
        user_friendly_names yes
        path_selector "round-robin 0"
}
multipaths {
        multipath {
        wwid "360002ac0000000000000002900014f1f"
        alias prox1
        path_grouping_policy multibus
        path_selector "round-robin 0"
        }
}
I've no idea why, but now after deleting the the two aliases in the bindings file, they don't appear anymore. I've also commented all the lines in multipath.conf and then restarted the service just to make sure it's not related to that, but the lines haven't reappeared and the pve name is back to the default WWID. But I'll take this behaviour as it is. What I'm interested in is why does multipath -v3 -ll say that /dev/sda is whitelisted after I clearly blacklisted its wwid?
[EDIT] After rebooting the whole system, it finally said that /dev/sda node named blacklisted. So I'm guessing that simply restarting the service (multipathd) is not enough.
[AFTER A WHILE] I rebooted it again and I get this:
Code:
root@pve:~# pvs
  WARNING: PV ITPy26-m4J2-GBW1-rJSr-5rdk-Wr09-j4r6k2 on /dev/sdc was already found on /dev/sdb.
  WARNING: PV ITPy26-m4J2-GBW1-rJSr-5rdk-Wr09-j4r6k2 on /dev/sdd was already found on /dev/sdb.
  WARNING: PV ITPy26-m4J2-GBW1-rJSr-5rdk-Wr09-j4r6k2 on /dev/sde was already found on /dev/sdb.
  WARNING: PV ITPy26-m4J2-GBW1-rJSr-5rdk-Wr09-j4r6k2 prefers device /dev/sdb because device was seen first.
  WARNING: PV ITPy26-m4J2-GBW1-rJSr-5rdk-Wr09-j4r6k2 prefers device /dev/sdb because device was seen first.
  WARNING: PV ITPy26-m4J2-GBW1-rJSr-5rdk-Wr09-j4r6k2 prefers device /dev/sdb because device was seen first.
  PV         VG       Fmt  Attr PSize   PFree
  /dev/sda3  pve      lvm2 a--  223.28g  15.85g
  /dev/sdb   prox1_vg lvm2 a--  199.98g 824.00m
I've no idea what is going on. Of course, I don't have access to the storage anymore. /dev/mapper doesn't appear anymore
 
Old 01-19-2018, 10:06 AM   #17
vincix
Senior Member
 
Registered: Feb 2011
Distribution: Ubuntu, Centos
Posts: 1,158

Original Poster
Rep: Reputation: 87
The problem was related to this stanza which I hadn't mentioned
Code:
blacklist {
        devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st|dcssblk)[0-9]"
        devnode "^(td|hd|vd)[a-z]"
        devnode "^nvme"
        devnode "^cciss!c[0-9]d[0-9]*"
       devnode "^sd[a-z]"
}
I'll get back to it on Monday. Thanks!

[Later EDIT]
So the problem would seem obvious, i.e. I've blacklisted all /dev/sdX, including "b" to "e". Yet, I don't understand what is going on with /dev/sdb. Was it seen before multipath was able to blacklist it? What's going on here? How can I interpret the messages?

Last edited by vincix; 01-19-2018 at 10:37 AM.
 
Old 01-22-2018, 08:53 AM   #18
vincix
Senior Member
 
Registered: Feb 2011
Distribution: Ubuntu, Centos
Posts: 1,158

Original Poster
Rep: Reputation: 87
Related question: weren't you supposed to be able to actually blacklist nodes /dev/sdX in order to make sure that only /dev/mapper/name is accessible? How does this work exactly?
 
Old 01-22-2018, 10:04 AM   #19
MensaWater
LQ Guru
 
Registered: May 2005
Location: Atlanta Georgia USA
Distribution: Redhat (RHEL), CentOS, Fedora, CoreOS, Debian, FreeBSD, HP-UX, Solaris, SCO
Posts: 7,830
Blog Entries: 15

Rep: Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668
The blacklist tells it which devices not to check to be part of multipath definitions. Not which devices ARE multipath.

If you told it to exclude /dev/sd* it would exclude the component disks that are part of your multipath.

Also relying on /dev/sd* names isn't valid as they can change on boot. If you blacklist /dev/sda and that is always the boot disk that has no other paths it wouldn't hurt but if on a boot for whatever reason it discovered your boot disk as /dev/sdb and your actual multipath as /dev/sda (which can happen) it would mean you didn't have multipath to the SAN disk and also couldn't use your boot disk. (Theoretically - since the boot happens before multipathing starts it might boot but other partitions on that boot disk might not work.)

It is much better to find a vendor (e.g. PERC, LSI or AVAGO internal RAID controller) or WWID of specific devices to blacklist as that will insure it only blacklists what you intend rather than dealing with the vagaries of SCSI /dev/sd* naming assignments.

If you install and run lsscsi it should show what each of your sd* devices are in relation to vendor. For example on one my servers I see my boot disk (/dev/sda) is presented by an AVAGO internal RAID controller:
[10:2:0:0] disk AVAGO MR9363-4i 4.65 /dev/sda

In my multipath.conf I blacklist that (and PERC as most of my servers have that). We also blacklist device from FusionIO (now SanDisk) by name rather than vendor because the name doesn't change on each boot like /dev/sd* devices:
Code:
blacklist {
wwid 26353900f02796769
devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
devnode "^hd[a-z]"
#
device {
           vendor "DELL"
           product "PERC*"
}
#
device {
           vendor "AVAGO"
           product "MR*"
}
#
devnode "^fio[a-z]"     #Fusion IO
}
Notice that even though my boot disk is /dev/sda I do not blacklist /dev/sda at all. The above AVAGO definition would blacklist the device if it were /dev/sda or if it were any other /dev/sd* such as /dev/sdq.
 
1 members found this post helpful.
Old 01-23-2018, 08:42 AM   #20
vincix
Senior Member
 
Registered: Feb 2011
Distribution: Ubuntu, Centos
Posts: 1,158

Original Poster
Rep: Reputation: 87
If I try to blacklist /dev/sda (blacklisting its wwid simply doesn't work: multipath -v3 -ll | grep sda gives: sda: udev property ID_WWN whitelisted; it only works by blacklisting with devnode). The weird thing is that, when I blacklisted /dev/sda, it said that it couldn't get device 253:3 (which is pve-data-tmeta - which I'm guessing is related to the thin pool that I created), and, as far as I could tell, if I don't blacklist it, then it's back to 253:15 which doesn't exist Now I blacklisted it again and the 253:3 error didn't occur anymore. Should I be worried about these inconsistencies? The LVM seems to be working fine, at a first glance.

Last edited by vincix; 01-23-2018 at 08:51 AM.
 
Old 01-23-2018, 12:33 PM   #21
MensaWater
LQ Guru
 
Registered: May 2005
Location: Atlanta Georgia USA
Distribution: Redhat (RHEL), CentOS, Fedora, CoreOS, Debian, FreeBSD, HP-UX, Solaris, SCO
Posts: 7,830
Blog Entries: 15

Rep: Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668
You seem to only do part of what I suggest or tell me only part of what you do so it isn't clear to me.

What does lsscsi show?

What does scsi_id with whitelist flag show for /dev/sda? Give your exact command line used.

When you say you "tried to blacklist" do you mean within multipath.conf? If so what does the entire blacklist section look like? I notice you earlier listed a blacklist stanza without this particular WWID. When you tried to blacklist by WWID did you do it within that existing stanza as I cautioned before or did you try to create a separate blacklist section?
 
Old 01-24-2018, 02:24 AM   #22
vincix
Senior Member
 
Registered: Feb 2011
Distribution: Ubuntu, Centos
Posts: 1,158

Original Poster
Rep: Reputation: 87
Sorry, I actually did use scsi_id after you said that I needed to explicitly whitelist it first, but there's nothing special to it. It just displays what I've already found out, the WWID of /dev/sda. Obviously, /dev/sdb to /dev/sde have the same wwid as /dev/mapper/prox1, which is the LUN WWID.
Code:
/lib/udev/scsi_id -g /dev/sda
3600508b1001ca273b75d7358da14e8e3
Code:
lsscsi -sd
[0:0:0:0]    storage HP       P220i            6.00  -               -
[0:1:0:0]    disk    HP       LOGICAL VOLUME   6.00  /dev/sda [8:0]   240GB
[1:0:0:0]    disk    3PARdata VV               3215  /dev/sdb [8:16]   214GB
[1:0:0:254]  enclosu 3PARdata SES              3215  -               -
[1:0:2:0]    disk    3PARdata VV               3215  /dev/sdc [8:32]   214GB
[1:0:2:254]  enclosu 3PARdata SES              3215  -               -
[2:0:0:0]    disk    3PARdata VV               3215  /dev/sdd [8:48]   214GB
[2:0:0:254]  enclosu 3PARdata SES              3215  -               -
[2:0:2:0]    disk    3PARdata VV               3215  /dev/sde [8:64]   214GB
[2:0:2:254]  enclosu 3PARdata SES              3215  -               -

Last edited by vincix; 01-24-2018 at 02:26 AM.
 
Old 01-24-2018, 08:15 AM   #23
MensaWater
LQ Guru
 
Registered: May 2005
Location: Atlanta Georgia USA
Distribution: Redhat (RHEL), CentOS, Fedora, CoreOS, Debian, FreeBSD, HP-UX, Solaris, SCO
Posts: 7,830
Blog Entries: 15

Rep: Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668
Based on that you should be able to blacklist (in multipath.conf) by doing:
Code:
device {
           vendor "HP"
           product "LOGICAL*"
}
Again you'd want to incorporate that within your existing blacklist stanza.

Doing it that way it would blacklist the device regardless of its /dev/sd* name (or any other name). Also if you later added another internal disk it would automatically get blacklisted.

By the way your output shows the major/minor of sda is 8/0.
 
1 members found this post helpful.
Old 01-25-2018, 04:36 AM   #24
vincix
Senior Member
 
Registered: Feb 2011
Distribution: Ubuntu, Centos
Posts: 1,158

Original Poster
Rep: Reputation: 87
Thanks for all the answers and patience. I'll mark it as solved, despite the awkward error, as the subject of the thread clearly was solved
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] finding out WWID of LUN in linux vincix Linux - Server 1 01-10-2018 11:59 AM
WWID udev rules bravored Linux - Enterprise 2 12-07-2016 01:52 PM
wwid of /dev/cciss/c0d0 yashraj221087 Linux - Server 4 09-06-2015 02:29 AM
[SOLVED] dm-n name and wwid canopus69 Linux - Server 3 12-15-2013 06:40 PM
cant find disk controller FreeBSD* ICH5 sata controller pele_smk *BSD 1 01-17-2010 03:45 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Server

All times are GMT -5. The time now is 06:28 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration