LinuxQuestions.org
Latest LQ Deal: Latest LQ Deals
Go Back   LinuxQuestions.org > Forums > Enterprise Linux Forums > Linux - Enterprise
User Name
Password
Linux - Enterprise This forum is for all items relating to using Linux in the Enterprise.

Notices


Reply
  Search this Thread
Old 01-25-2017, 10:41 AM   #1
Hsingh
LQ Newbie
 
Registered: Jan 2012
Location: CO , USA
Distribution: Redhat, Oracle Linux6, Ubuntu16 & CentOS.
Posts: 28

Rep: Reputation: Disabled
SAN LUNs not using multipath


Our database server, running Oracle Linux 6.4 has 12 SAN LUNs(Hitachi) presented. But only 8 show up as multipath devices and, remaining 4 show up as local scsi disks, which means these LUNs have no redundant paths.

Output of PVS :
Code:
  PV                   VG               Fmt  Attr PSize   PFree
  /dev/mapper/mpathap1 vg01             lvm2 a--  725.00g    0 
  /dev/mapper/mpathbp1 vg02             lvm2 a--  150.00g    0 
  /dev/mapper/mpathcp1 vg03             lvm2 a--   34.99g    0 
  /dev/mapper/mpathdp1 vg04             lvm2 a--   59.99g    0 
  /dev/mapper/mpathep1 vg05             lvm2 a--   34.99g    0 
  /dev/mapper/mpathfp1 vg06             lvm2 a--  229.99g    0 
  /dev/mapper/mpathhp1 vg07             lvm2 a--  229.99g    0 
  /dev/mapper/mpathip1 vg08             lvm2 a--   50.00g    0 
  /dev/sdav1           vg07             lvm2 a--  270.00g    0 
  /dev/sdba2           vg_denplpilatdb1 lvm2 a--  247.62g 5.44g
  /dev/sdbl1           vg02             lvm2 a--  105.00g    0 
  /dev/sdbz1           vg02             lvm2 a--  110.00g    0 
  /dev/sdca1           vg07             lvm2 a--  200.00g    0


After doing some research, only solution I found is making changes to /etc/multipath.conf. Specifically comment out the parameter {wwid "*"} under blacklist and then restart the multipathd services. Not very sure if I should make this change since this is a heavily used server ~200+ users.

content of /etc/multipath.conf:
Code:
# multipath.conf written by anaconda

defaults {
        user_friendly_names yes
}
blacklist {
        devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
        devnode "^hd[a-z]"
        devnode "^dcssblk[0-9]*"
        device {
                vendor "DGC"
                product "LUNZ"
        }
        device {
                vendor "IBM"
                product "S/390.*"
        }
        # don't count normal SATA devices as multipaths
        device {
                vendor  "ATA"
        }
        # don't count 3ware devices as multipaths
        device {
                vendor  "3ware"
        }
        device {
                vendor  "AMCC"
        }
        # nor highpoint devices
        device {
                vendor  "HPT"
        }
        wwid "3600508b1001c9c5986be0c6503c1857e"
        device {
                vendor HP
                product Virtual_DVD-ROM
        }
        wwid "*"


Has anyone seen a similar issue or know of a fix? Is this even as big of a problem or can be ignored ?
 
Old 01-25-2017, 02:14 PM   #2
MensaWater
LQ Guru
 
Registered: May 2005
Location: Atlanta Georgia USA
Distribution: Redhat (RHEL), CentOS, Fedora, CoreOS, Debian, FreeBSD, HP-UX, Solaris, SCO
Posts: 7,831
Blog Entries: 15

Rep: Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668
The "pvs" command only shows the physical volumes already made LVM Physical Volumes. That is to say if you've not yet done a pvcreate on the new devices it doesn't show them.

Not all disks on the system are necessarily multipath (e.g. internal disks, dedicated arrays with their own HBAs, SSD HBAs etc...).

The output that would be helpful is "lsscsi" as it would show you all the SCSI devices including disks along with types. (You may need to "yum install lsscsi" package to get the lsscsi command.)

You would also want to run "multipath -l -v2" or similar to see what Multipath actually sees. It should show you the multipath device (in your case mpath...) as well as any component scsi disk (sd...) paths for all the paths that make up that multipath device.

Typically when doing pvcreate I specify the /dev/mapper/<multipath> device (/dev/mapper/mpath... in your case. If you specified the scsi disk (/dev/sd...) instead it may restrict it to that though normally LVM will find the same PV info on all paths. You may need to look at the "filter" in your /etc/lvm.conf to verify it can see all mpath... devices if the sd... devices you listed are in fact from the array and there are multiple paths to them as shown by lsscsi.

You don't say which Hitachi you have. Another tool I've found very useful from Hitachi comes with their HORCM software and it is called "inqraid". If you have that you can run a command to see the relationship between specific Hitachi devices and sd... devices by running:
ls /dev/sd* |inqraid -CLI -fxng
NOTE: That will also include non-Hitachi sd... devices if you have them but will show "-" for all fields other than the first one for those non-Hitachi devices.

You can determine settings for /etc/multipath.conf by reviewing array documentation for your array. On RHEL6 for Hitachi VSP the definition we use here is:
Code:
      device {
               vendor                  "(HITACHI|HP)"
               product                 "OPEN-.*"
               getuid_callout          "/lib/udev/scsi_id --whitelisted --device=/dev/%n"
              features                "1 queue_if_no_path"
               hardware_handler        "0"
               path_selector           "round-robin 0"
               path_grouping_policy    multibus
               rr_weight               uniform
               rr_min_io               100
               path_checker            tur
               prio                    const
       }
NOTE: Make sure that is inside your "devices section" below one of the other types defined and above the terminating bracket for the section.
Also after any edit of multipath.conf run "service multipathd restart" so the daemon re-reads it. (This can be done on a running system with no impact to existing devices.)

By the way - what you posted wasn't the "devices" section but rather the "blacklist" section. Do NOT add your array to the "blacklist" section.

You don't mention if you ran a rescan to detect new devices. You'd have to have done that (or a reboot) after presenting them from the disk array. A nice tool to do that is rescan-scsi-bus.sh which you can get by installing sg3_utils package with yum.

Last edited by MensaWater; 01-25-2017 at 03:13 PM.
 
1 members found this post helpful.
Old 01-25-2017, 04:08 PM   #3
Hsingh
LQ Newbie
 
Registered: Jan 2012
Location: CO , USA
Distribution: Redhat, Oracle Linux6, Ubuntu16 & CentOS.
Posts: 28

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by MensaWater View Post
The "pvs" command only shows the physical volumes already made LVM Physical Volumes. That is to say if you've not yet done a pvcreate on the new devices it doesn't show them.
I know for a fact that we have done fdisk/pvcreate on all the SAN devices


Quote:
Not all disks on the system are necessarily multipath (e.g. internal disks, dedicated arrays with their own HBAs, SSD HBAs etc...).
Correct, but there are only two local devices, everyhing else is coming from SAN


Quote:
The output that would be helpful is "lsscsi" as it would show you all the SCSI devices including disks along with types. (You may need to "yum install lsscsi" package to get the lsscsi command.)
Code:
 lsscsi
[0:0:0:0]    disk    HITACHI  OPEN-V           7303  /dev/sda 
[0:0:0:1]    disk    HITACHI  OPEN-V           7303  /dev/sdb 
[0:0:0:2]    disk    HITACHI  OPEN-V           7303  /dev/sdc 
[0:0:0:3]    disk    HITACHI  OPEN-V           7303  /dev/sdd 
[0:0:0:4]    disk    HITACHI  OPEN-V           7303  /dev/sde 
[0:0:0:5]    disk    HITACHI  OPEN-V           7303  /dev/sdf 
[0:0:0:6]    disk    HITACHI  OPEN-V           7303  /dev/sdg 
[0:0:0:7]    disk    HITACHI  OPEN-V           7303  /dev/sdh 
[0:0:0:8]    disk    HITACHI  OPEN-V           7303  /dev/sdi 
[0:0:0:9]    disk    HITACHI  OPEN-V           7303  /dev/sdj 
[0:0:0:10]   disk    HITACHI  OPEN-V           7303  /dev/sdk 
[0:0:0:11]   disk    HITACHI  OPEN-V           7303  /dev/sdl 
[0:0:0:12]   disk    HITACHI  OPEN-V           7303  /dev/sdm 
[0:0:1:0]    disk    HITACHI  OPEN-V           7303  /dev/sdn 
[0:0:1:1]    disk    HITACHI  OPEN-V           7303  /dev/sdo 
[0:0:1:2]    disk    HITACHI  OPEN-V           7303  /dev/sdp 
[0:0:1:3]    disk    HITACHI  OPEN-V           7303  /dev/sdq 
[0:0:1:4]    disk    HITACHI  OPEN-V           7303  /dev/sdr 
[0:0:1:5]    disk    HITACHI  OPEN-V           7303  /dev/sds 
[0:0:1:6]    disk    HITACHI  OPEN-V           7303  /dev/sdt 
[0:0:1:7]    disk    HITACHI  OPEN-V           7303  /dev/sdu 
[0:0:1:8]    disk    HITACHI  OPEN-V           7303  /dev/sdv 
[0:0:1:9]    disk    HITACHI  OPEN-V           7303  /dev/sdw 
[0:0:1:10]   disk    HITACHI  OPEN-V           7303  /dev/sdx 
[0:0:1:11]   disk    HITACHI  OPEN-V           7303  /dev/sdy 
[0:0:1:12]   disk    HITACHI  OPEN-V           7303  /dev/sdz 
[0:0:2:0]    disk    HITACHI  OPEN-V           7303  /dev/sdaa
[0:0:2:1]    disk    HITACHI  OPEN-V           7303  /dev/sdab
[0:0:2:2]    disk    HITACHI  OPEN-V           7303  /dev/sdac
[0:0:2:3]    disk    HITACHI  OPEN-V           7303  /dev/sdad
[0:0:2:4]    disk    HITACHI  OPEN-V           7303  /dev/sdae
[0:0:2:5]    disk    HITACHI  OPEN-V           7303  /dev/sdaf
[0:0:2:6]    disk    HITACHI  OPEN-V           7303  /dev/sdag
[0:0:2:7]    disk    HITACHI  OPEN-V           7303  /dev/sdah
[0:0:2:8]    disk    HITACHI  OPEN-V           7303  /dev/sdai
[0:0:2:9]    disk    HITACHI  OPEN-V           7303  /dev/sdaj
[0:0:2:10]   disk    HITACHI  OPEN-V           7303  /dev/sdak
[0:0:2:11]   disk    HITACHI  OPEN-V           7303  /dev/sdal
[0:0:2:12]   disk    HITACHI  OPEN-V           7303  /dev/sdam
[0:0:3:0]    disk    HITACHI  OPEN-V           7303  /dev/sdan
[0:0:3:1]    disk    HITACHI  OPEN-V           7303  /dev/sdao
[0:0:3:2]    disk    HITACHI  OPEN-V           7303  /dev/sdap
[0:0:3:3]    disk    HITACHI  OPEN-V           7303  /dev/sdaq
[0:0:3:4]    disk    HITACHI  OPEN-V           7303  /dev/sdar
[0:0:3:5]    disk    HITACHI  OPEN-V           7303  /dev/sdas
[0:0:3:6]    disk    HITACHI  OPEN-V           7303  /dev/sdat
[0:0:3:7]    disk    HITACHI  OPEN-V           7303  /dev/sdau
[0:0:3:8]    disk    HITACHI  OPEN-V           7303  /dev/sdav
[0:0:3:9]    disk    HITACHI  OPEN-V           7303  /dev/sdaw
[0:0:3:10]   disk    HITACHI  OPEN-V           7303  /dev/sdax
[0:0:3:11]   disk    HITACHI  OPEN-V           7303  /dev/sday
[0:0:3:12]   disk    HITACHI  OPEN-V           7303  /dev/sdaz
[1:0:0:0]    disk    HITACHI  OPEN-V           7303  /dev/sdbb
[1:0:0:1]    disk    HITACHI  OPEN-V           7303  /dev/sdbc
[1:0:0:2]    disk    HITACHI  OPEN-V           7303  /dev/sdbd
[1:0:0:3]    disk    HITACHI  OPEN-V           7303  /dev/sdbe
[1:0:0:4]    disk    HITACHI  OPEN-V           7303  /dev/sdbf
[1:0:0:5]    disk    HITACHI  OPEN-V           7303  /dev/sdbg
[1:0:0:6]    disk    HITACHI  OPEN-V           7303  /dev/sdbh
[1:0:0:7]    disk    HITACHI  OPEN-V           7303  /dev/sdbi
[1:0:0:8]    disk    HITACHI  OPEN-V           7303  /dev/sdbj
[1:0:0:9]    disk    HITACHI  OPEN-V           7303  /dev/sdbk
[1:0:0:10]   disk    HITACHI  OPEN-V           7303  /dev/sdbl
[1:0:0:11]   disk    HITACHI  OPEN-V           7303  /dev/sdbm
[1:0:0:12]   disk    HITACHI  OPEN-V           7303  /dev/sdbn
[1:0:1:0]    disk    HITACHI  OPEN-V           7303  /dev/sdbo
[1:0:1:1]    disk    HITACHI  OPEN-V           7303  /dev/sdbp
[1:0:1:2]    disk    HITACHI  OPEN-V           7303  /dev/sdbq
[1:0:1:3]    disk    HITACHI  OPEN-V           7303  /dev/sdbr
[1:0:1:4]    disk    HITACHI  OPEN-V           7303  /dev/sdbs
[1:0:1:5]    disk    HITACHI  OPEN-V           7303  /dev/sdbt
[1:0:1:6]    disk    HITACHI  OPEN-V           7303  /dev/sdbu
[1:0:1:7]    disk    HITACHI  OPEN-V           7303  /dev/sdbv
[1:0:1:8]    disk    HITACHI  OPEN-V           7303  /dev/sdbw
[1:0:1:9]    disk    HITACHI  OPEN-V           7303  /dev/sdbx
[1:0:1:10]   disk    HITACHI  OPEN-V           7303  /dev/sdby
[1:0:1:11]   disk    HITACHI  OPEN-V           7303  /dev/sdbz
[1:0:1:12]   disk    HITACHI  OPEN-V           7303  /dev/sdca
[1:0:2:0]    disk    HITACHI  OPEN-V           7303  /dev/sdcb
[1:0:2:1]    disk    HITACHI  OPEN-V           7303  /dev/sdcc
[1:0:2:2]    disk    HITACHI  OPEN-V           7303  /dev/sdcd
[1:0:2:3]    disk    HITACHI  OPEN-V           7303  /dev/sdce
[1:0:2:4]    disk    HITACHI  OPEN-V           7303  /dev/sdcf
[1:0:2:5]    disk    HITACHI  OPEN-V           7303  /dev/sdcg
[1:0:2:6]    disk    HITACHI  OPEN-V           7303  /dev/sdch
[1:0:2:7]    disk    HITACHI  OPEN-V           7303  /dev/sdci
[1:0:2:8]    disk    HITACHI  OPEN-V           7303  /dev/sdcj
[1:0:2:9]    disk    HITACHI  OPEN-V           7303  /dev/sdck
[1:0:2:10]   disk    HITACHI  OPEN-V           7303  /dev/sdcl
[1:0:2:11]   disk    HITACHI  OPEN-V           7303  /dev/sdcm
[1:0:2:12]   disk    HITACHI  OPEN-V           7303  /dev/sdcn
[1:0:3:0]    disk    HITACHI  OPEN-V           7303  /dev/sdco
[1:0:3:1]    disk    HITACHI  OPEN-V           7303  /dev/sdcp
[1:0:3:2]    disk    HITACHI  OPEN-V           7303  /dev/sdcq
[1:0:3:3]    disk    HITACHI  OPEN-V           7303  /dev/sdcr
[1:0:3:4]    disk    HITACHI  OPEN-V           7303  /dev/sdcs
[1:0:3:5]    disk    HITACHI  OPEN-V           7303  /dev/sdct
[1:0:3:6]    disk    HITACHI  OPEN-V           7303  /dev/sdcu
[1:0:3:7]    disk    HITACHI  OPEN-V           7303  /dev/sdcv
[1:0:3:8]    disk    HITACHI  OPEN-V           7303  /dev/sdcw
[1:0:3:9]    disk    HITACHI  OPEN-V           7303  /dev/sdcx
[1:0:3:10]   disk    HITACHI  OPEN-V           7303  /dev/sdcy
[1:0:3:11]   disk    HITACHI  OPEN-V           7303  /dev/sdcz
[1:0:3:12]   disk    HITACHI  OPEN-V           7303  /dev/sdda
[2:0:0:0]    disk    HP       LOGICAL VOLUME   4.68  /dev/sdba
[2:3:0:0]    storage HP       P220i            4.68  -
Quote:
You would also want to run "multipath -l -v2" or similar to see what Multipath actually sees. It should show you the multipath device (in your case mpath...) as well as any component scsi disk (sd...) paths for all the paths that make up that multipath device.
"multipath -l -v2" shows the same result of what I usually use "multipath -ll"

Code:
sudo multipath -l -v2
mpathe (360060e80132b960050202b9600000092) dm-2 HITACHI,OPEN-V
size=35G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=0 status=active
  |- 0:0:0:1  sdb  8:16   active undef running
  |- 0:0:1:1  sdo  8:224  active undef running
  |- 0:0:2:1  sdab 65:176 active undef running
  |- 0:0:3:1  sdao 66:128 active undef running
  |- 1:0:0:1  sdbc 67:96  active undef running
  |- 1:0:1:1  sdbp 68:48  active undef running
  |- 1:0:2:1  sdcc 69:0   active undef running
  `- 1:0:3:1  sdcp 69:208 active undef running
mpathd (360060e80132b960050202b9600000095) dm-1 HITACHI,OPEN-V
size=60G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=0 status=active
  |- 0:0:0:4  sde  8:64   active undef running
  |- 0:0:1:4  sdr  65:16  active undef running
  |- 0:0:2:4  sdae 65:224 active undef running
  |- 0:0:3:4  sdar 66:176 active undef running
  |- 1:0:0:4  sdbf 67:144 active undef running
  |- 1:0:1:4  sdbs 68:96  active undef running
  |- 1:0:2:4  sdcf 69:48  active undef running
  `- 1:0:3:4  sdcs 70:0   active undef running
mpathc (360060e80132b960050202b9600000093) dm-3 HITACHI,OPEN-V
size=35G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=0 status=active
  |- 0:0:0:2  sdc  8:32   active undef running
  |- 0:0:1:2  sdp  8:240  active undef running
  |- 0:0:2:2  sdac 65:192 active undef running
  |- 0:0:3:2  sdap 66:144 active undef running
  |- 1:0:0:2  sdbd 67:112 active undef running
  |- 1:0:1:2  sdbq 68:64  active undef running
  |- 1:0:2:2  sdcd 69:16  active undef running
  `- 1:0:3:2  sdcq 69:224 active undef running
mpathb (360060e80132b960050202b9600000094) dm-6 HITACHI,OPEN-V
size=255G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=0 status=active
  |- 0:0:0:3  sdd  8:48   active undef running
  |- 0:0:1:3  sdq  65:0   active undef running
  |- 0:0:2:3  sdad 65:208 active undef running
  |- 0:0:3:3  sdaq 66:160 active undef running
  |- 1:0:0:3  sdbe 67:128 active undef running
  |- 1:0:1:3  sdbr 68:80  active undef running
  |- 1:0:2:3  sdce 69:32  active undef running
  `- 1:0:3:3  sdcr 69:240 active undef running
mpatha (360060e80132b960050202b9600000091) dm-0 HITACHI,OPEN-V
size=725G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=0 status=active
  |- 0:0:0:0  sda  8:0    active undef running
  |- 0:0:1:0  sdn  8:208  active undef running
  |- 0:0:2:0  sdaa 65:160 active undef running
  |- 0:0:3:0  sdan 66:112 active undef running
  |- 1:0:0:0  sdbb 67:80  active undef running
  |- 1:0:1:0  sdbo 68:32  active undef running
  |- 1:0:2:0  sdcb 68:240 active undef running
  `- 1:0:3:0  sdco 69:192 active undef running
mpathi (360060e80132b960050202b96000000e0) dm-13 HITACHI,OPEN-V
size=50G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=0 status=active
  |- 0:0:0:7  sdh  8:112  active undef running
  |- 0:0:1:7  sdu  65:64  active undef running
  |- 0:0:2:7  sdah 66:16  active undef running
  |- 0:0:3:7  sdau 66:224 active undef running
  |- 1:0:0:7  sdbi 67:192 active undef running
  |- 1:0:1:7  sdbv 68:144 active undef running
  |- 1:0:2:7  sdci 69:96  active undef running
  `- 1:0:3:7  sdcv 70:48  active undef running
mpathh (360060e80132b960050202b96000000a9) dm-14 HITACHI,OPEN-V
size=230G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=0 status=active
  |- 0:0:0:6  sdg  8:96   active undef running
  |- 0:0:1:6  sdt  65:48  active undef running
  |- 0:0:2:6  sdag 66:0   active undef running
  |- 0:0:3:6  sdat 66:208 active undef running
  |- 1:0:0:6  sdbh 67:176 active undef running
  |- 1:0:1:6  sdbu 68:128 active undef running
  |- 1:0:2:6  sdch 69:80  active undef running
  `- 1:0:3:6  sdcu 70:32  active undef running
mpathf (360060e80132b960050202b9600000096) dm-4 HITACHI,OPEN-V
size=230G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=0 status=active
  |- 0:0:0:5  sdf  8:80   active undef running
  |- 0:0:1:5  sds  65:32  active undef running
  |- 0:0:2:5  sdaf 65:240 active undef running
  |- 0:0:3:5  sdas 66:192 active undef running
  |- 1:0:0:5  sdbg 67:160 active undef running
  |- 1:0:1:5  sdbt 68:112 active undef running
  |- 1:0:2:5  sdcg 69:64  active undef running
  `- 1:0:3:5  sdct 70:16  active undef running

Code:
Typically when doing pvcreate I specify the /dev/mapper/<multipath> device (/dev/mapper/mpath... in your case.   If you specified the scsi disk (/dev/sd...) instead it may restrict it to that though normally LVM will find the same PV info on all paths.   You may need to look at the "filter" in your /etc/lvm.conf to verify it can see all mpath... devices if the sd... devices you listed are in fact from the array and there are multiple paths to them as shown by lsscsi.
I think that could be the problem that we used pvcreate as /dev/sd and not /dev/mapper/. Is there a way to redo that ? I havent checked the filter in /etc/lvm.conf , but doing it now.

Quote:
You don't say which Hitachi you have
HUSVM

Quote:
You can determine settings for /etc/multipath.conf by reviewing array documentation for your array.
I would think if it was config problem, wouldnt it effect all the LUNs ? To your earlier point, I think when we did pvcreate we used /dev/sd instead of /dev/mapper/xx

Quote:
By the way - what you posted wasn't the "devices" section but rather the "blacklist" section. Do NOT add your array to the "blacklist" section.
ROGER THAT !

Quote:
You don't mention if you ran a rescan to detect new devices. You'd have to have done that (or a reboot) after presenting them from the disk array. A nice tool to do that is rescan-scsi-bus.sh which you can get by installing sg3_utils package with yum.
Have used "rescan-scsi-bus.sh" and its a hit or miss. It worked sometimes and, other times we have to reboot to see new LUNs.

Last edited by Hsingh; 01-25-2017 at 04:12 PM. Reason: edit2
 
Old 01-25-2017, 04:14 PM   #4
Hsingh
LQ Newbie
 
Registered: Jan 2012
Location: CO , USA
Distribution: Redhat, Oracle Linux6, Ubuntu16 & CentOS.
Posts: 28

Original Poster
Rep: Reputation: Disabled
And thanks a lot for replying ! Sorry my last response got pretty long.
I am worried that the devices which show up as /dev/sd , do not have redundant paths to storage, is that concern even legit ?
 
Old 01-25-2017, 04:56 PM   #5
MensaWater
LQ Guru
 
Registered: May 2005
Location: Atlanta Georgia USA
Distribution: Redhat (RHEL), CentOS, Fedora, CoreOS, Debian, FreeBSD, HP-UX, Solaris, SCO
Posts: 7,831
Blog Entries: 15

Rep: Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668
The lsscsi confirms your devices, sdav, sdbl, sdbz and sdca are Hitachi OPEN-V devices. Comparing the ID on those devices to those you see for your other multipath devices suggests each is a single path to a separate Hitachi device.

The multipath output confirms that you have 8 paths to most of your devices but doesn't show anything containing these 4 devices so I suspect your issue is simply that multipath hasn't updated to create multipath devices yet.
As I noted before you can run "service multipathd restart". This bounces the daemon. Stopping/bouncing the daemon doesn't remove or impact any active multipath devices. On restart the daemon should find the new devices.

Based on the pattern I see in your multipath output it seems you should have a multipath that contains 8 paths for each of the 4 devices you saw in pvs but only one of each is showing in your pvs output yet all of which are in your lsscsi output. For example your first disk has 8 paths in lsscsi but only the 1 in pvs:
0:0:0:8
0:0:1:8
0:0:2:8
0:0:3:8 sdav In pvs output
1:0:0:8
1:0:1:8
1:0:2:8
1:0:3:8

As noted above I believe your host has already re-scanned and is seeing the new devices so you can ignore what follows but I did want to say more about rescan.

The rescan command that I mentioned has worked well for me in recent months but there are other ways to rescan.

One way is to do an echo into each of your host adapters. Based on the output you gave you appear to have host0 and host1 as adapters as denoted by the lines that start "0:" or "1:". (You appear to have multiple ports from the disk array zoned to the server given that you show 0:0: through 0:3: and 1:0: through 1:3: in your multipath for other devices giving each disk 8 paths in total.)

The command to rescan that way is:
echo "- - -" >host0/scan
echo "- - -" >host1/scan

Here we use Qlogic HBAs - you can determine which Qlogic hosts you have by going to /sys/class/scsi_host and running:
grep -i ql */* 2>/dev/null
That will find any that have host#s that have QL part numbers.

Also Qlogic provides a tool we install on our servers which includes a rescan script they provide called:
ql-dynamic-tgt-lun-disc.sh
If you're running Qlogic you can download that from their site as part of the Super-Installer package. (When I last did this searching by OS found it where searching by Qlogic model didn't on their site.)
If you're using something else (e.g. Emulex) you may find a similar tool on that vendor's site.

I've not had to reboot a server in years to see new devices because one of these methods always works. As noted above the first rescan tool I posted about is one I've used multiple times on both RHEL5 and RHEL6 systems. Your OEL is ported from RHEL though Oracle does add things. I'd expect these things to work.

Last edited by MensaWater; 01-26-2017 at 10:25 AM.
 
1 members found this post helpful.
Old 01-27-2017, 12:16 PM   #6
Hsingh
LQ Newbie
 
Registered: Jan 2012
Location: CO , USA
Distribution: Redhat, Oracle Linux6, Ubuntu16 & CentOS.
Posts: 28

Original Poster
Rep: Reputation: Disabled
Wink

Sorry about the delayed response. Had to get an approval for even a non-disruptive change. I got an OK to restart the multipathd daemon for this weekend.

You are correct, we have multiple ports zoned in, using Brocade SAN fabrics.
And we are using Emulex HBAs, see below. I checked their website but haven't found a re-scan utility/script yet but, I will keep looking.

I checked the history and, confirmed that I have used the echo command in past, see below.

Code:
03:00.0 Fibre Channel: Emulex Corporation Saturn-X: LightPulse Fibre Channel Host Adapter (rev 03)
03:00.1 Fibre Channel: Emulex Corporation Saturn-X: LightPulse Fibre Channel Host Adapter (rev 03)

echo "- - -" >  /sys/class/scsi_host/host0/scan
echo "- - -" >  /sys/class/scsi_host/host1/scan
Quote:
Based on the pattern I see in your multipath output it seems you should have a multipath that contains 8 paths for each of the 4 devices you saw in pvs but only one of each is showing in your pvs output yet all of which are in your lsscsi output. For example your first disk has 8 paths in lsscsi but only the 1 in pvs:
0:0:0:8
0:0:1:8
0:0:2:8
0:0:3:8 sdav In pvs output
1:0:0:8
1:0:1:8
1:0:2:8
1:0:3:8
I am confused about what you are saying here.
When I search for these 4 devices (sdav, sdbl, sdbz and sdca) in lsscsi output, I only see it once but, you are saying its there with 8 paths ?

Also for full disclosure, I also see duplicate path warnings. Which apparently I can write a reg ex filter for in /etc/lvm/lvm.conf and ignore.
Code:
Found duplicate PV cllQixBtG36oy39iVPdzmj8GFAhtX7ww: using /dev/sday1 not /dev/sdbm1
  Found duplicate PV onG5ThUCXuBr60rT1W6fIjIOGjSKNaNk: using /dev/sdaz1 not /dev/sdbn1
  Found duplicate PV OGHvsLvxJ1yrRGyPHl2dFaKVSwQoa1BG: using /dev/sdak1 not /dev/sdax1
  Found duplicate PV YE4PrCVMvsdfhddQHtefTj0MzXSkcdpO: using /dev/sdcw1 not /dev/sdai1
  Found duplicate PV YE4PrCVMvsdfhddQHtefTj0MzXSkcdpO: using /dev/sdv1 not /dev/sdcw1
  Found duplicate PV cllQixBtG36oy39iVPdzmj8GFAhtX7ww: using /dev/sdal1 not /dev/sday1
  Found duplicate PV onG5ThUCXuBr60rT1W6fIjIOGjSKNaNk: using /dev/sdam1 not /dev/sdaz1
  Found duplicate PV OGHvsLvxJ1yrRGyPHl2dFaKVSwQoa1BG: using /dev/sdcy1 not /dev/sdak1
  Found duplicate PV OGHvsLvxJ1yrRGyPHl2dFaKVSwQoa1BG: using /dev/sdx1 not /dev/sdcy1
  Found duplicate PV YE4PrCVMvsdfhddQHtefTj0MzXSkcdpO: using /dev/sdcj1 not /dev/sdv1
  Found duplicate PV cllQixBtG36oy39iVPdzmj8GFAhtX7ww: using /dev/sdcz1 not /dev/sdal1
  Found duplicate PV YE4PrCVMvsdfhddQHtefTj0MzXSkcdpO: using /dev/sdi1 not /dev/sdcj1
  Found duplicate PV cllQixBtG36oy39iVPdzmj8GFAhtX7ww: using /dev/sdy1 not /dev/sdcz1
  Found duplicate PV onG5ThUCXuBr60rT1W6fIjIOGjSKNaNk: using /dev/sdda1 not /dev/sdam1
  Found duplicate PV onG5ThUCXuBr60rT1W6fIjIOGjSKNaNk: using /dev/sdz1 not /dev/sdda1
  Found duplicate PV OGHvsLvxJ1yrRGyPHl2dFaKVSwQoa1BG: using /dev/sdcl1 not /dev/sdx1
  Found duplicate PV OGHvsLvxJ1yrRGyPHl2dFaKVSwQoa1BG: using /dev/sdk1 not /dev/sdcl1
  Found duplicate PV YE4PrCVMvsdfhddQHtefTj0MzXSkcdpO: using /dev/sdbw1 not /dev/sdi1
  Found duplicate PV cllQixBtG36oy39iVPdzmj8GFAhtX7ww: using /dev/sdcm1 not /dev/sdy1
  Found duplicate PV cllQixBtG36oy39iVPdzmj8GFAhtX7ww: using /dev/sdl1 not /dev/sdcm1
  Found duplicate PV onG5ThUCXuBr60rT1W6fIjIOGjSKNaNk: using /dev/sdcn1 not /dev/sdz1
  Found duplicate PV onG5ThUCXuBr60rT1W6fIjIOGjSKNaNk: using /dev/sdm1 not /dev/sdcn1
  Found duplicate PV OGHvsLvxJ1yrRGyPHl2dFaKVSwQoa1BG: using /dev/sdby1 not /dev/sdk1
  Found duplicate PV YE4PrCVMvsdfhddQHtefTj0MzXSkcdpO: using /dev/sdbj1 not /dev/sdbw1
  Found duplicate PV cllQixBtG36oy39iVPdzmj8GFAhtX7ww: using /dev/sdbz1 not /dev/sdl1
  Found duplicate PV onG5ThUCXuBr60rT1W6fIjIOGjSKNaNk: using /dev/sdca1 not /dev/sdm1
  Found duplicate PV YE4PrCVMvsdfhddQHtefTj0MzXSkcdpO: using /dev/sdav1 not /dev/sdbj1
  Found duplicate PV OGHvsLvxJ1yrRGyPHl2dFaKVSwQoa1BG: using /dev/sdbl1 not /dev/sdby1
 
Old 01-27-2017, 01:07 PM   #7
MensaWater
LQ Guru
 
Registered: May 2005
Location: Atlanta Georgia USA
Distribution: Redhat (RHEL), CentOS, Fedora, CoreOS, Debian, FreeBSD, HP-UX, Solaris, SCO
Posts: 7,831
Blog Entries: 15

Rep: Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668
Quote:
Originally Posted by Hsingh View Post
I am confused about what you are saying here.
When I search for these 4 devices (sdav, sdbl, sdbz and sdca) in lsscsi output, I only see it once but, you are saying its there with 8 paths ?
You are correct - in the example I gave sdav itself appears only once in lsscsi output. Since an sd name is not, itself, a multipath, it would only appear once under that name. However, other sd names would appear for other paths to the same SAN disk (or LUN or LDEV) in a multipath environment.

I was saying that while we can see the I/O path for sdav is 0:0:3:8 in your lsscsi output. We can also see 7 other I/O paths that are *likely* to be separate paths to the same SAN disk (LUN/LDEV). Those being:
0:0:0:8
0:0:1:8
0:0:2:8
1:0:0:8
1:0:1:8
1:0:2:8
1:0:3:8
I did not list the sd* names associated with each of those because I'm lazy and you can see them yourself in your lsscsi output.

The reason I say they are *likely* paths to the same disk is the pattern I see in your other multipath disks as reported by multipath -ll (multipath -l -v2) suggests them. For example the mpathe you show contains the following:
|- 0:0:0:1 sdb 8:16 active undef running
|- 0:0:1:1 sdo 8:224 active undef running
|- 0:0:2:1 sdab 65:176 active undef running
|- 0:0:3:1 sdao 66:128 active undef running
|- 1:0:0:1 sdbc 67:96 active undef running
|- 1:0:1:1 sdbp 68:48 active undef running
|- 1:0:2:1 sdcc 69:0 active undef running
`- 1:0:3:1 sdcp 69:208 active undef running

That suggests the common item in IO path to each of the 8 sds for mpathe is the last field of the colon separate IO path is 1 for each of those meaning the first 3 fields are defining the path and the last field is defining the specific disk (LUN/LDEV). So your 8 IO paths (without specific final field) are:
0:0:0:x
0:0:1:x
0:0:2:x
0:0:3:x
1:0:0:x
1:0:1:x
1:0:2:x
1:0:3:x

So for mpathe final character on each of those is "1". For mpathd the final character on each of those is "4".

For your device with name sdav as shown in lsscsi the final character is "8" which is why I suggested the other 7 IO paths I did each starting as shown above but ending with "8".

It wasn't clear from what you wrote whether you actually ran the "service multipathd restart" after you got permission. If you did then you should rerun "multipath -l -v2" (or multipath -ll) to see if it now shows your multipath devices for the other 4 SAN disks (LUNS/LDEVS) each containing 8 component sd devices (4 x 8 = 32 sd devices overall for the 4 mpaths created).

NOTE:
On reboot both "sd" names and "mpath" names can change as the names are assigned as devices are found during each boot. Examples:
What is sdf on this boot might become sdam on another boot.
What is mpathk on this boot might become mpathb on another boot.

Last edited by MensaWater; 01-27-2017 at 01:40 PM.
 
1 members found this post helpful.
Old 01-28-2017, 01:05 AM   #8
Hsingh
LQ Newbie
 
Registered: Jan 2012
Location: CO , USA
Distribution: Redhat, Oracle Linux6, Ubuntu16 & CentOS.
Posts: 28

Original Poster
Rep: Reputation: Disabled
Have not yet ran "service multipathd restart".

I am afraid the theory of last field value being the LUN ID may not apply.
We have total of 12 LUNs mapped. But the lsscsi output have values ranging from 0:0:0:0 - 0:0:0:12, which would mean the OS is seeing total of 13 LUNs, it doesnt add up.

You are right about sd names possibly changing, but I think mptah names are static, since they are defined in /dev/mapper.

Code:
ls -la /dev/mapper/
total 0
drwxr-xr-x.  2 root root     580 Aug 12 11:11 .
drwxr-xr-x. 27 root root    9640 Aug 12 11:47 ..
crw-rw----.  1 root root 10, 236 Aug 12 11:10 control
lrwxrwxrwx.  1 root root       7 Aug 12 11:11 mpatha -> ../dm-0
lrwxrwxrwx.  1 root root       7 Aug 12 11:11 mpathap1 -> ../dm-5
lrwxrwxrwx.  1 root root       7 Aug 12 11:11 mpathb -> ../dm-6
lrwxrwxrwx.  1 root root       8 Aug 12 11:11 mpathbp1 -> ../dm-11
lrwxrwxrwx.  1 root root       7 Aug 12 11:11 mpathc -> ../dm-3
lrwxrwxrwx.  1 root root       7 Aug 12 11:11 mpathcp1 -> ../dm-9
lrwxrwxrwx.  1 root root       7 Aug 12 11:11 mpathd -> ../dm-1
lrwxrwxrwx.  1 root root       7 Aug 12 11:11 mpathdp1 -> ../dm-7
lrwxrwxrwx.  1 root root       7 Aug 12 11:11 mpathe -> ../dm-2
lrwxrwxrwx.  1 root root       7 Aug 12 11:11 mpathep1 -> ../dm-8
lrwxrwxrwx.  1 root root       7 Aug 12 11:11 mpathf -> ../dm-4
lrwxrwxrwx.  1 root root       8 Aug 12 11:11 mpathfp1 -> ../dm-10
lrwxrwxrwx.  1 root root       8 Aug 12 11:11 mpathh -> ../dm-14
lrwxrwxrwx.  1 root root       8 Aug 12 11:11 mpathhp1 -> ../dm-15
lrwxrwxrwx.  1 root root       8 Aug 12 11:11 mpathi -> ../dm-13
lrwxrwxrwx.  1 root root       8 Aug 12 11:11 mpathip1 -> ../dm-16
lrwxrwxrwx.  1 root root       8 Aug 12 11:11 vg01-oradata -> ../dm-24
lrwxrwxrwx.  1 root root       8 Aug 12 11:11 vg02-archive -> ../dm-25
lrwxrwxrwx.  1 root root       8 Aug 12 11:11 vg03-redolog1 -> ../dm-21
lrwxrwxrwx.  1 root root       8 Aug 12 11:11 vg04-oracle -> ../dm-23
lrwxrwxrwx.  1 root root       8 Aug 12 11:11 vg05-redolog2 -> ../dm-22
lrwxrwxrwx.  1 root root       8 Aug 12 11:11 vg06-backups -> ../dm-20
lrwxrwxrwx.  1 root root       8 Aug 12 11:11 vg07-fast_recovery_area -> ../dm-19
lrwxrwxrwx.  1 root root       8 Aug 12 11:11 vg08-sybase -> ../dm-18
lrwxrwxrwx.  1 root root       8 Aug 12 11:11 vg_denplpilatdb1-home -> ../dm-17
lrwxrwxrwx.  1 root root       8 Aug 12 11:10 vg_denplpilatdb1-root -> ../dm-12
 
Old 01-30-2017, 08:55 AM   #9
MensaWater
LQ Guru
 
Registered: May 2005
Location: Atlanta Georgia USA
Distribution: Redhat (RHEL), CentOS, Fedora, CoreOS, Debian, FreeBSD, HP-UX, Solaris, SCO
Posts: 7,831
Blog Entries: 15

Rep: Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668
I feel confident the pattern I mentioned is correct. If you notice, all 104 lsscsi lines that start with 0: or 1: (your 2 controllers) ending in :0 through :12 show as Hitachi OPEN-V regardless of first character. This means you do (or did) have 13 Hitachi devices. 13 devices x 8 paths = 104. A couple of reasons you'd have more than you think are:
1) You had another device presented at one time and removed from the Hitachi side but didn't do cleanup from the server side.
2) You have another device presented that you're not taking into account. Not all devices are used for filesystems (or LVM). You might have another for various reasons such as:
a) Hitachi command device - These are used by Hitachi operations such as Shadow Image or Copy On Write - the server has to see them even though they don't actually store any OS level detail.
b) Raw or block device used by an application. Oracle ASM for example uses raw/block devices for database storage rather than using filesystems such as ext4. From OS level you don't see the data files but they are there managed by ASM.

You could run a command like "lsblk" or "fdisk -luc" on each of the sd devices shown to determine full details of each. Its possible one or more would give you errors indicating the device doesn't really exist and if so can be removed but I suspect they do all exist.

A more certain way to verify this would be to go into the Hitachi array's management page itself, locate the host HBAs (on VSP these are defined as "host groups" - not sure if that is the same in your Hitachi) to see how many LUNs (on VSP these are defined as LDEVs) are presented to those HBAs. Make sure when looking at them you don't have any "filters" turned on that might be doing something like excluding "CMD" devices or others.

Finally as I've said several times running "service multipathd restart" does NOT affect currently active multipath devices - it simply makes it rediscover to see any new ones. I've run this command dozens of times on multiple servers and have never seen it cause an issue. Even turning off with "service multipathd stop" won't affect currently active multipath devices.

Last edited by MensaWater; 02-01-2017 at 01:29 PM.
 
Old 02-03-2017, 12:26 PM   #10
Hsingh
LQ Newbie
 
Registered: Jan 2012
Location: CO , USA
Distribution: Redhat, Oracle Linux6, Ubuntu16 & CentOS.
Posts: 28

Original Poster
Rep: Reputation: Disabled
Didn't mean to abandon this thread. I still have not been able to get on OK to work on this.
Will keep this discussion open and update asap.
 
Old 02-10-2017, 08:28 AM   #11
MensaWater
LQ Guru
 
Registered: May 2005
Location: Atlanta Georgia USA
Distribution: Redhat (RHEL), CentOS, Fedora, CoreOS, Debian, FreeBSD, HP-UX, Solaris, SCO
Posts: 7,831
Blog Entries: 15

Rep: Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668
Quote:
Originally Posted by Hsingh View Post
Didn't mean to abandon this thread. I still have not been able to get on OK to work on this.
Will keep this discussion open and update asap.
Any update?
 
Old 02-17-2017, 09:40 AM   #12
Hsingh
LQ Newbie
 
Registered: Jan 2012
Location: CO , USA
Distribution: Redhat, Oracle Linux6, Ubuntu16 & CentOS.
Posts: 28

Original Poster
Rep: Reputation: Disabled
Not yet. I should have an update by early next week.
 
Old 02-21-2017, 11:03 AM   #13
Hsingh
LQ Newbie
 
Registered: Jan 2012
Location: CO , USA
Distribution: Redhat, Oracle Linux6, Ubuntu16 & CentOS.
Posts: 28

Original Poster
Rep: Reputation: Disabled
Quick question, since I am doing multipathd restart, would this be a good time to make change to "/etc/multipath.conf"? It was suggested that we should try removing the asterisk from the blacklist option, and that might enable all the LUNs to be in multi path.

Anything I need to worried about when making changes to multipath.conf? Thanks!

Code:
# multipath.conf written by anaconda

defaults {
	user_friendly_names yes
}
blacklist {
	devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
	devnode "^hd[a-z]"
	devnode "^dcssblk[0-9]*"
	device {
		vendor "DGC"
		product "LUNZ"
	}
	device {
		vendor "IBM"
		product "S/390.*"
	}
	# don't count normal SATA devices as multipaths
	device {
		vendor  "ATA"
	}
	# don't count 3ware devices as multipaths
	device {
		vendor  "3ware"
	}
	device {
		vendor  "AMCC"
	}
	# nor highpoint devices
	device {
		vendor  "HPT"
	}
	wwid "3600508b1001c9c5986be0c6503c1857e"
	device {
		vendor HP
		product Virtual_DVD-ROM
	}
	wwid "*"
}
 
Old 02-21-2017, 01:00 PM   #14
MensaWater
LQ Guru
 
Registered: May 2005
Location: Atlanta Georgia USA
Distribution: Redhat (RHEL), CentOS, Fedora, CoreOS, Debian, FreeBSD, HP-UX, Solaris, SCO
Posts: 7,831
Blog Entries: 15

Rep: Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668Reputation: 1668
I've never used * for the wwid in the blacklist of any multipath.conf.

Review of the man page says for blacklist:
Quote:
The blacklist section is used to exclude specific device from inclusion
in the multipath topology. It is most commonly used to exclude local
disks or LUNs for the array controller.

The following keywords are recognized:

wwid The World Wide Identification of a device.

devnode Regular expression of the device nodes to be excluded.

device Subsection for the device description. This subsection
recognizes the vendor and product keywords. For a full
description of these keywords please see the devices
section description.
The fact it mentions regular expressions for devnode (where * is used for some entries) but not for wwid suggests the * metacharacter does nothing in wwid specification.

The fact you have existing mpaths despite the existence of the * metacharacter further seems to confirm it is ignoring that and finding the Hitachi devices. This means having the line probably isn't hurting or helping so leaving it there likely won't cause you a problem but were it me I'd probably remove it just to avoid future confusion.
 
Old 02-27-2017, 11:30 AM   #15
Hsingh
LQ Newbie
 
Registered: Jan 2012
Location: CO , USA
Distribution: Redhat, Oracle Linux6, Ubuntu16 & CentOS.
Posts: 28

Original Poster
Rep: Reputation: Disabled
Doing the change tomorrow after hours. Plan is to edit the multupath.conf and, then restart multipathd.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Multipath San Disk identification pantdk Linux - Server 5 01-02-2013 10:17 AM
Need help with best configuration for SAN multipath kameleon25 Linux - General 1 05-12-2012 02:55 AM
Using multipath for disk created on SAN codenjanod Linux - Software 1 07-27-2010 12:57 PM
Mapping LUNs to multipath devices. larold Linux - Enterprise 9 02-20-2010 11:31 AM
autosense SAN LUNS sg001 Linux - Server 1 07-17-2007 10:10 AM

LinuxQuestions.org > Forums > Enterprise Linux Forums > Linux - Enterprise

All times are GMT -5. The time now is 10:16 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration