Only seeing one SAN path

aayore · 07-02-2010, 09:41 AM

I have a server (RHEL 5.5, 2.6.18-92.1.6.el5) connected to two SANs by multiple paths. But the server's only seeing one path at a time. The only exceptions are the controllers from my HP EVA SAN, as you can see below (which indicates correct zoning). Sometimes when I reboot the server, everything will switch to the other path, but it has never (that I know of) switched while the server is up and running.

[root@cobra ~]# lsscsi
[0:0:0:1] disk NETAPP LUN 7320 /dev/sda
[0:0:0:2] disk NETAPP LUN 7320 /dev/sdb
[0:0:2:0] storage HP HSV210 5000 -
[0:0:2:6] disk HP HSV210 5000 /dev/sdc
[0:0:2:7] disk HP HSV210 5000 /dev/sdd
[0:0:2:8] disk HP HSV210 5000 /dev/sde
[0:0:2:9] disk HP HSV210 5000 /dev/sdf
[0:0:3:0] storage HP HSV210 5000 -
[1:0:2:0] storage HP HSV210 6220 -
[1:0:3:0] storage HP HSV210 6220 -
[root@cobra ~]#

I have triple-checked the zoning and masking/presentation, and everything looks correct. I'm working this through a paid support contract, too, but they're proving useless. Any ideas?

anomie · 07-02-2010, 12:56 PM

How are you intending for the multipathing to work? Have you configured dm multipath?

aayore · 07-02-2010, 03:01 PM

My multipath configuration is below, but I think the root problem is at a lower level. I should be seeing more devices on my HBA's. There aren't currently multiple paths for multipathd to multipath.

The following is the multipath.conf that I'm using. It's the one recommended by NetApp and the one that I'm using on most of my other servers (and working fine there).

Code:

[root@cobra ~]# cat /etc/multipath.conf
defaults
{
    user_friendly_names     yes
    max_fds                 4096
    queue_without_daemon    no
}
blacklist
{
    wwid DevId
    devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
    devnode "^hd[a-z]"
    devnode "^cciss!c[0-9]d[0-9]*[p[0-9]*]"
}
devices
{
    device
     {
        vendor                  "NETAPP"
        product                 "LUN"
        getuid_callout          "/sbin/scsi_id -g -u -s /block/%n"
        prio_callout            "/sbin/mpath_prio_ontap /dev/%n"
        features                "1 queue_if_no_path"
        hardware_handler        "0"
        path_grouping_policy    group_by_prio
        failback                immediate
        rr_weight               uniform
        rr_min_io               128
        path_checker            directio
        flush_on_last_del       yes
        }
}
[root@cobra ~]#

Multipath is up and running, but it only sees one path.

Code:

[root@cobra ~]# multipath -ll
mpath9 (360a98000572d5071614a5744742d324d) dm-0 NETAPP,LUN
[size=2.0T][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=1][active]
 \_ 0:0:0:1 sda 8:0   [active][ready]
mpath8 (3600508b40010881d0001900000350000) dm-5 HP,HSV210
[size=500G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=50][active]
 \_ 0:0:2:9 sdf 8:80  [active][ready]
mpath7 (3600508b40010881d0001300003aa0000) dm-4 HP,HSV210
[size=500G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=50][active]
 \_ 0:0:2:8 sde 8:64  [active][ready]
mpath6 (3600508b40010881d0001300003790000) dm-3 HP,HSV210
[size=500G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=50][active]
 \_ 0:0:2:7 sdd 8:48  [active][ready]
mpath5 (3600508b40010881d0001300003330000) dm-2 HP,HSV210
[size=500G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=50][active]
 \_ 0:0:2:6 sdc 8:32  [active][ready]
mpath10 (360a98000572d50514e34583354714539) dm-1 NETAPP,LUN
[size=240G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=4][active]
 \_ 0:0:0:2 sdb 8:16  [active][ready]
[root@cobra ~]#

anomie · 07-02-2010, 03:41 PM

Please put configs / code output in code tags. (See my sig.)

Has the multipath.conf you posted above ever worked on the borked host? IOW, were you seeing the correct number of paths at any point, and then things stopped working?

Also, is a Brocade switch connecting host to SAN? (If so, have you run diagnostics there?) How difficult would it be to temporarily "open things up" for zoning?

Given all the details you have supplied, and depending on the answers to the above, this sounds like it could be a physical issue.

aayore · 07-02-2010, 03:53 PM

This server has never (to my knowledge) seen multiple paths. It's been running for a while (a few months) with single paths, and I'm trying to get multipath set up for the first time.

I have tried different cables, different SFP's, and different ports on the switch (yes - Brocade). I wouldn't rule out a hardware issue completely, but it's weird that both paths work just fine - I just have to reboot to switch from one path to the other. I've tried to open up the zoning as much as possible, but based on the fact that I can get both paths to work without changing any zoning/masking/presentation, I don't know what else to check in that area. I haven't run diagnostics there yet, but if you think it's worthwhile, I can follow that lead.

anomie · 07-02-2010, 09:34 PM

Given that multipath(8) is 1) not seeing more than one path; 2) not reporting any errors, I think it would be a good idea to see what you can learn from the switch. I don't have my Brocade documentation handy, but I believe you can view logs and other info using "supportsave" (or some such).

Wish I had a better lead for you. I will say that I've bumped into some rather vexing issues with Brocade switches, where (even after replacing SFPs and fibre cables) the entire unit had to be replaced to resolve the problem. In that case, though, the symptoms were manifesting themselves in obvious ways. There were Brocade switch events in the logs, and the ports would go offline and then mysteriously come back online.