Hi guys

I have 1 server HP use RedHat 5.7 with 1 HBA (2 port). Each port connect to 1 SAN switch. And SAN switch connect to HP P2000 G3 FC storage.

I use multipath to make 2 fiber link failover each other. But when i plug out 1 link, it s not as i expected. Server cant connect to Storage anymore, need restart to server use the other link.

Can someone give me some advice to fix this issue?
Thank you very much!!!

Here is the output of /var/log/message:
Jul 7 18:39:11 db2-pri kernel: sd 2:0:0:0: SCSI error: return code = 0x00010000
Jul 7 18:39:11 db2-pri kernel: end_request: I/O error, dev sda, sector 720977
Jul 7 18:39:15 db2-pri multipathd: sdb: tur checker reports path is down
Jul 7 18:39:15 db2-pri multipathd: sda: tur checker reports path is down
Jul 7 18:39:15 db2-pri multipathd: sdb: tur checker reports path is down
Jul 7 18:39:16 db2-pri kernel: sd 2:0:0:0: SCSI error: return code = 0x00010000
Jul 7 18:39:16 db2-pri kernel: end_request: I/O error, dev sda, sector 720977
Jul 7 18:39:20 db2-pri multipathd: sda: tur checker reports path is down
Jul 7 18:39:20 db2-pri multipathd: sdb: tur checker reports path is down
Jul 7 18:39:20 db2-pri kernel: sd 2:0:0:0: SCSI error: return code = 0x00010000
Jul 7 18:39:20 db2-pri kernel: end_request: I/O error, dev sda, sector 720977
Jul 7 18:39:24 db2-pri kernel: sd 2:0:0:0: SCSI error: return code = 0x00010000
Jul 7 18:39:24 db2-pri kernel: end_request: I/O error, dev sda, sector 720977

and here is my config file:

#blacklist {
# devnode "*"
#}


defaults {
user_friendly_names yes
# features 0
}

multipaths {
multipath {
wwid 3600c0ff0001a37f043c78c5301000000
alias my

}
}

devices {
device {
vendor "HP"
product "P2000 G3 FC"
path_grouping_policy group_by_prio
getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
prio_callout "/sbin/mpath_prio_alua /dev/%n"
path_selector "round-robin 0"
path_checker tur
hardware_handler "0"
failback immediate
rr_weight uniform
rr_min_io 100
no_path_retry 2
}
}

Output of command: multipath -ll

my (3600c0ff0001a37f043c78c5301000000) dm-2 HP,P2000 G3 FC
[size=121G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=100][active]
\_ 2:0:0:0 sda 8:0 [active][ready]
\_ 3:0:0:0 sdc 8:32 [active][ready]
\_ round-robin 0 [prio=20][enabled]
\_ 2:0:1:0 sdb 8:16 [active][ready]
\_ 3:0:1:0 sdd 8:48 [active][ready]

In your multipath -ll output I see 4 paths to the disk instead of the 2 suggested by your 2 HBA ports to 1 SAN switch. This suggests you've configured 2 zones (one for each HBA port) that each contain BOTH HBA ports.

Your log shows that 2 of the paths (/dev/sda and /dev/sdb) went down but the other 2 paths (/dev/sdc and /dev/sdd) did NOT go down so it appears to me multipathing itself should have worked.

However, you do NOT say what issue you saw when you unplugged the cable.

I suspect your issue is that you were NOT using the multipath device but rather just one of its /dev/sd* components.

Since your alias is "my" any use of the multipath disk must be using /dev/mpath/my or /dev/mapper/my. If instead you used one of the /dev/sd* paths you aren't actually using the multipath - you're just using one of its components.
So for example if you create an LVM Volume Group with "vgcreate" the disk you should specify would be /dev/mapper/my NOT /dev/sda or any other /dev/sd* device.) The same would be true of any meta disk (software raid) you created or if you were using a partition. (That is if you had a /dev/sda1 using it would not be using multipathing - you'd have to have something like /dev/mapper/my1.)

Hi,

I read that as each port is connected to a separate FC switch - I'd assume that each switch is connected to both controllers, hence the 4 paths.

trung1490 - I suspect that the rest of MensaWater's assessment is correct, however I have seen multipathd not fail over as expected when using an incorrect configuration e.g. incorrect path checker methods. I'd check the documentation provided by HP for your storage array.

Cheers,
Steve

Hi Mensa

Thank you for your responding.

I have 2 SAN switch and my Storage have 2 controller (active/active). In Storage, port 1 of controller 1 and controller 2 connect to SAN switch 1; port 2 of both controller connect to SAN switch 2. My server connect to 2 SAN switch.
I just create 1 volume in storage and map it to 2 port of my server. And all of these sda,sdb, sdc, sdd acctually just 1 volume.

When i plug out 1 link, output of multipath -ll:

[root@db2-pri trungnq]# multipath -ll
sda: checker msg is "tur checker reports path is down"
sdb: checker msg is "tur checker reports path is down"
my (3600c0ff0001a37f043c78c5301000000) dm-2 HP,P2000 G3 FC
[size=121G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=50][active]
\_ 2:0:0:0 sda 8:0 [active][faulty]
\_ 3:0:0:0 sdc 8:32 [active][ready]
\_ round-robin 0 [prio=10][enabled]
\_ 2:0:1:0 sdb 8:16 [active][faulty]
\_ 3:0:1:0 sdd 8:48 [active][ready]

Output when i plug out 1 fiber link in /var/log/message:

Jul 7 20:06:43 db2-pri kernel: BFA[0000:07:00.0][error] BFA_AEN_PORT_DISCONNECT: Base port (WWN = 10:00:8c:7c:ff:65:d1:6a) lost fabric connectivity.

Jul 7 20:07:14 db2-pri kernel: rport-2:0-0: blocked FC remote port time out: saving binding
Jul 7 20:07:14 db2-pri kernel: rport-2:0-1: blocked FC remote port time out: saving binding
Jul 7 20:07:14 db2-pri kernel: sd 2:0:0:0: SCSI error: return code = 0x00010000
Jul 7 20:07:14 db2-pri kernel: end_request: I/O error, dev sda, sector 721489
Jul 7 20:07:14 db2-pri kernel: sd 2:0:0:0: SCSI error: return code = 0x00010000
Jul 7 20:07:14 db2-pri kernel: end_request: I/O error, dev sda, sector 720977
Jul 7 20:07:15 db2-pri kernel: device-mapper: multipath: Failing path 8:0.
Jul 7 20:07:15 db2-pri multipathd: sda: tur checker reports path is down
Jul 7 20:07:15 db2-pri multipathd: checker failed path 8:0 in map my
Jul 7 20:07:15 db2-pri multipathd: my: remaining active paths: 3
Jul 7 20:07:15 db2-pri multipathd: sdb: tur checker reports path is down
Jul 7 20:07:15 db2-pri kernel: device-mapper: multipath: Failing path 8:16.
Jul 7 20:07:15 db2-pri multipathd: checker failed path 8:16 in map my
Jul 7 20:07:15 db2-pri multipathd: my: remaining active paths: 2
Jul 7 20:07:15 db2-pri multipathd: dm-2: add map (uevent)
Jul 7 20:07:15 db2-pri multipathd: dm-2: devmap already registered
Jul 7 20:07:15 db2-pri multipathd: dm-2: add map (uevent)
Jul 7 20:07:15 db2-pri multipathd: dm-2: devmap already registered
Jul 7 20:07:15 db2-pri kernel: sd 2:0:0:0: SCSI error: return code = 0x00010000
Jul 7 20:07:15 db2-pri kernel: end_request: I/O error, dev sda, sector 720977
Jul 7 20:07:19 db2-pri kernel: sd 2:0:0:0: SCSI error: return code = 0x00010000
Jul 7 20:07:19 db2-pri kernel: end_request: I/O error, dev sda, sector 720977
Jul 7 20:07:20 db2-pri multipathd: sda: tur checker reports path is down
Jul 7 20:07:20 db2-pri multipathd: sdb: tur checker reports path is down
Jul 7 20:07:24 db2-pri kernel: sd 2:0:0:0: SCSI error: return code = 0x00010000
Jul 7 20:07:24 db2-pri kernel: end_request: I/O error, dev sda, sector 720977
Jul 7 20:07:25 db2-pri multipathd: sda: tur checker reports path is down
Jul 7 20:07:25 db2-pri multipathd: sdb: tur checker reports path is down

I want to make these link failover to make redundancy connection to storage. And I use /dev/mpath/my (1 2 3) to mount into my server. My target is when 1 link fail, server still connect, read, write data to storage with smalles downtime.

Thank you

TrungNQ

Hi Steve

As i see in var/log/mess; multipath already dectected link down but i dont understand why it still have IO error; im quite mess about this. I will try other path checker: readsector0, directio.


And additional information about my server:

[root@db2-pri ~]# rpm -qa | grep multipath
device-mapper-multipath-0.4.7-46.el5
[root@db2-pri ~]# uname -r
2.6.18-274.el5
[root@db2-pri ~]# cat /etc/*-release
Red Hat Enterprise Linux Server release 5.7 (Tikanga)

Thank you

TrungNQ

Your output confirms that sda and sdb are "faulty" when you unplug the one cable but that sdc and sdd are still "ready" so again i say it appears to me multipathing is working as designed.

You still haven't said what issue you're seeing when you unplug the cable. If you're truly using the /dev/mpath/my (1 2 3) as you indicated then they should still work assuming you didn't just copy, link or rename one the /dev/sd (1 2 3) devices. By (1 2 3) I'm assuming you mean /dev/mpath/my1, /dev/mpath/my2 and /dev/mpath/my3. Can you confirm that?

The major and minor number of a device specify what it actually is more than the name so if you copied, linked or moved from one of the /dev/sd (1 2 3) devices to the /dev/mpath ( 1 2 3) you're not using the multipath device. Note that /dev/mpath contains symbolic links usually to the /dev/dm* (dm-2 in your case) devices. The /dev/dm* devices and the /dev/mapper/* devices will show you the major/minor for the multipath devices. You can see the major/minor on files with ls -l.

So for example you might see:
ls -l /dev/sda
brw-r----- 1 root disk 8, 0 May 25 02:32 /dev/sda
ls -l /dev/sdb
brw-r----- 1 root disk 8, 16 May 25 02:32 /dev/sdb
ls -l /dev/sdc
brw-r----- 1 root disk 8, 32 May 25 02:32 /dev/sdc
ls -l /dev/sdd
brw-r----- 1 root disk 8, 48 May 25 02:32 /dev/sdd

ls -l /dev/mpath/my
lrwxrwxrwx 1 root root 8 May 25 06:34 /dev/mpath/my -> ../dm-2

ls -l /dev/dm-2
brw-rw---- 1 root root 253, 2 May 25 06:34 /dev/dm-2

ls -l /dev/mapper/my
brw-rw---- 1 root root 253, 2 May 25 06:34 /dev/dm-2

Notice that /dev/mpath/my is a symbolic link to dm-2 in the above. All other entries however are block devices (as noted by the leading "b"). For all the devices other than the link you see the major and minor separated by comma just before the date/time stamp.

So you see that /dev/dm-2 and /dev/mapper/my have the SAME major/minor of "253, 2". However that is DIFFERENT than the major/minor numbers on each of the component /dev/sd* devices which are "8, 0", "8, 16", "8, 32" and "8, 48" respectively. So /dev/sda is the "8, 0" and typically the minor is incremented by 1 for each partition there so /dev/sda1 would be "8, 1", /dev/sda2 would be "8, 2" etc...
If you had a /dev/mappery/my2 with "8, 2" you'd really be pointing to the component partition rather than the multipath partition. The partition for the multipath device would have major 253 like the main disk but the minor would be different. I'm not sure how the minors are assigned in RHEL5 though for multipath devices since I don't use partitions myself. From the one RHEL6 system I have it appears the minors are assigned in sequence with the other disks as they are created. Do you have /dev/mapper/my* devices with major that matches that of /dev/mapper/my itself (e.g. major 253) but separate minors (e.g. /dev/mapper/my has minor 2 but my1 has a number other than 2)?

Hi,

I wouldn't just try these as failover has been detected but I would confirm you have the correct configuration as per your storage vendor's recommendations. If you can't find it, raise a ticket and ask HP - after all you pay them good money for support! (You do have support, don't you?)

Please could you post the output of:
Cheers,
Steve

You DID have an IO error on the devices (/dev/sda and /dev/sdb) that went away. However, the point in multipathing is that the IO would then retry on your alternate paths (/dev/sdc and /dev/sdd). Notice that the IO error you saw in the log specifies the device.

Hi Steve

I will provide my /etc/fstab for you as soon as possible. But i didnt manually edit that file.


To Mensa: Yes, i use /dev/mpath/my1; /dev/mpath/my2; /dev/mpath/my3 to mount file system into it. My problem is: when i plug out 1 link; my server lose connection to storage, it s not swap to the second link. If i restart server,my server can connect to storage over second link.

After edit multipath.conf; i use fdisk /dev/dm-2 to create new partitions. 1 of my friend told me that i should: fdisk /dev/mapper/my rather than i did. I guess the reason is dm-* is not stable, it can change in the next reboot...Do i understand it right?

Some document recommend use /dev/mapper/... rather than /dev/mpath/...; do you know why? And i will check major and minnor number of each component when i come back office tommorow.


Once again, thank you very much for all of your support!

Best regards,

TrungNQ

In RHEL5 both /dev/mpath and /dev/mapper (as well as /dev/dm*) exist. In RHEL6 /dev/mpath went away so that may be why you see some documents talking about only /dev/mapper.

The /dev/dm* devices are ephemeral like the /dev/sd* devices. That is to say the physical disk that is /dev/sdg this time around might be /dev/sda or some other /dev/sd* after a reboot. Similarly what is /dev/dm-2 this time around might be /dev/dm-7 or some other /dev/dm-* device after a reboot. Because of this it is typically recommended you not use these in your definitions. Depending on what you do there are ways to modify (e.g. use of labels rather than devices in fstab, use of /dev/mapper devices is LVM etc...). Also some things will find whatever you're using because of their nature (e.g. LVM actually scans all disks to see if there are LVM structures and will find the disk no matter what name it has on a reboot.)

There IS an alternate method called "user friendly names" that would define the devices as mpath* (e.g. /dev/mpath/mpath5, /dev/mapper/mpath5) but these are also ephemeral so what is mpath5 on this boot might be mpath2 or some other mpath* next boot.

When you hard code the alias like you did in multipath.conf as "my" the WWID you specified is unique so this name does NOT change on each reboot and this is one of the main reasons for using such aliases.

Until you've verified the major/minor on the devices you're using however you're using them there is no way to guarantee you have the multipath devices.

You seem to confirm you're using partitions but then say you haven't edited anything in fstab so the question is exactly what are you doing. (e.g. are you using LVM? meta-disks? Some other more esoteric method such as ASM devices for Oracle RAC/GRID?)

You say it didn't fail over but in fact your logs suggest that multipath only failed the link you pulled which means the other link IS there with associated components so the question is: WHAT did not failover? That is did your VG not failover? Did your Meta-disk not failover? Did your ASM not failover?

What you're doing with the disks and/or partitions dictates other information. A disk all by itself (even in multipath) does nothing except wait to be used in some way and so far you've not told us how you're using it.

Hi Mensa

I use multipath device for ASM disk groups. (for Oracle RAC)
/dev/mpath/my1 mapped to OCRVote
/dev/mpath/my2 mapped to FileRecovery
/dev/mpath/my3 mapped to DATAFILE

About the your question: what didnt failover? It s me ASM disk group, when i plug out 1 link, server cant use DB (read, write) and start Oracle Instance anymore.

And this is information about my partitions:

======================================================
[root@db1-pri ~]# cd /dev/mapper/
[root@db1-pri mapper]# ls -al
total 0
drwxr-xr-x 2 root root 180 Jul 7 18:48 .
drwxr-xr-x 18 root root 4220 Jul 7 18:49 ..
crw------- 1 root root 10, 63 Jul 7 18:47 control
brw-rw---- 1 root disk 253, 2 Jul 7 18:48 my
brw-rw---- 1 root disk 253, 3 Jul 7 18:48 myp1
brw-rw---- 1 root disk 253, 4 Jul 7 18:48 myp2
brw-rw---- 1 root disk 253, 5 Jul 7 18:48 myp3
brw-rw---- 1 root disk 253, 0 Jul 7 18:48 VolGroup00-LogVol00
brw-rw---- 1 root disk 253, 1 Jul 7 18:47 VolGroup00-LogVol01

================================================================

[root@db1-pri mapper]# cd /dev/mpath/
[root@db1-pri mpath]# ls -al
total 0
drwxr-xr-x 2 root root 120 Jul 7 18:48 .
drwxr-xr-x 18 root root 4220 Jul 7 18:49 ..
lrwxrwxrwx 1 root root 7 Jul 7 18:48 my -> ../dm-2
lrwxrwxrwx 1 root root 7 Jul 7 18:48 myp1 -> ../dm-3
lrwxrwxrwx 1 root root 7 Jul 7 18:48 myp2 -> ../dm-4
lrwxrwxrwx 1 root root 7 Jul 7 18:48 myp3 -> ../dm-5

===============================================================

[root@db1-pri mpath]# ls -al /dev/sda
brw-r----- 1 root disk 8, 0 Jul 7 18:47 /dev/sda
[root@db5-pri ~]# ls -al /dev/sda1
brw-r----- 1 root disk 8, 1 Jul 7 14:39 /dev/sda1
[root@db5-pri ~]# ls -al /dev/sda2
brw-r----- 1 root disk 8, 2 Jul 7 14:39 /dev/sda2
[root@db5-pri ~]# ls -al /dev/sda2
brw-r----- 1 root disk 8, 2 Jul 7 14:39 /dev/sda2

================================================================

[root@db1-pri mpath]# ls -al /dev/dm-2
brw-rw---- 1 root root 253, 2 Jul 7 18:48 /dev/dm-2
[root@db1-pri mpath]# ls -al /dev/dm-3
brw-rw---- 1 root root 253, 3 Jul 7 18:48 /dev/dm-3
[root@db1-pri mpath]# ls -al /dev/dm-4
brw-rw---- 1 root root 253, 4 Jul 7 18:48 /dev/dm-4
[root@db1-pri mpath]# ls -al /dev/dm-5
brw-rw---- 1 root root 253, 5 Jul 7 18:48 /dev/dm-5

I tried to install 2 other server for Oracle RAC (named: cluster 2) but with difference mapping for ASM disk groups:
/dev/dm-3 mapped to OCRVote
/dev/dm-4 mapped to FileRecovery
/dev/dm-5 mapped to DATAFILE
(my bad, i forgot /dev/mapper/...)
After all of RAC installation, i plug out 1 link and everything still work normally, and in /var/log/message:

Jul 7 21:20:33 omap1-pri kernel: bfa 0000:07:00.0: Base port (WWN = 10:00:8c:7c:ff:65:d1:52) lost fabric connectivity
Jul 7 21:21:04 omap1-pri kernel: rport-2:0-1: blocked FC remote port time out: saving binding
Jul 7 21:21:04 omap1-pri kernel: rport-2:0-0: blocked FC remote port time out: saving binding
Jul 7 21:21:04 omap1-pri kernel: sd 2:0:1:0: SCSI error: return code = 0x00010000
Jul 7 21:21:04 omap1-pri kernel: end_request: I/O error, dev sdb, sector 720976
Jul 7 21:21:04 omap1-pri kernel: device-mapper: multipath: Failing path 8:16.
Jul 7 21:21:04 omap1-pri kernel: sd 2:0:1:0: SCSI error: return code = 0x00010000
Jul 7 21:21:04 omap1-pri kernel: end_request: I/O error, dev sdb, sector 721488
Jul 7 21:21:04 omap1-pri kernel: sd 2:0:1:0: SCSI error: return code = 0x00010000
Jul 7 21:21:04 omap1-pri kernel: end_request: I/O error, dev sdb, sector 9992577
Jul 7 21:21:04 omap1-pri kernel: sd 2:0:1:0: SCSI error: return code = 0x00010000
Jul 7 21:21:04 omap1-pri kernel: end_request: I/O error, dev sdb, sector 10196114
Jul 7 21:21:04 omap1-pri kernel: sd 2:0:1:0: SCSI error: return code = 0x00010000
Jul 7 21:21:04 omap1-pri kernel: end_request: I/O error, dev sdb, sector 61177185
Jul 7 21:21:04 omap1-pri kernel: sd 2:0:1:0: SCSI error: return code = 0x00010000
Jul 7 21:21:04 omap1-pri multipathd: dm-2: add map (uevent)
Jul 7 21:21:04 omap1-pri kernel: end_request: I/O error, dev sdb, sector 61185377
Jul 7 21:21:04 omap1-pri kernel: sd 2:0:1:0: SCSI error: return code = 0x00010000
Jul 7 21:21:04 omap1-pri kernel: end_request: I/O error, dev sdb, sector 61380722
Jul 7 21:21:04 omap1-pri multipathd: dm-2: devmap already registered
Jul 7 21:21:04 omap1-pri kernel: device-mapper: multipath: Failing path 8:0.
Jul 7 21:21:04 omap1-pri multipathd: 8:16: mark as failed
Jul 7 21:21:04 omap1-pri multipathd: my: remaining active paths: 3
Jul 7 21:21:04 omap1-pri multipathd: sda: tur checker reports path is down
Jul 7 21:21:04 omap1-pri multipathd: checker failed path 8:0 in map my
Jul 7 21:21:04 omap1-pri multipathd: my: remaining active paths: 2
Jul 7 21:21:04 omap1-pri multipathd: sdb: tur checker reports path is down
Jul 7 21:21:04 omap1-pri multipathd: dm-2: add map (uevent)
Jul 7 21:21:04 omap1-pri multipathd: dm-2: devmap already registered
Jul 7 21:21:14 omap1-pri multipathd: sda: tur checker reports path is down
Jul 7 21:21:14 omap1-pri multipathd: sdb: tur checker reports path is down
Jul 7 21:21:24 omap1-pri multipathd: sda: tur checker reports path is down
Jul 7 21:21:24 omap1-pri multipathd: sdb: tur checker reports path is down
Jul 7 21:21:34 omap1-pri multipathd: sda: tur checker reports path is down
Jul 7 21:21:34 omap1-pri multipathd: sdb: tur checker reports path is down
Jul 7 21:21:44 omap1-pri multipathd: sda: tur checker reports path is down
Jul 7 21:21:54 omap1-pri multipathd: sdb: tur checker reports path is down
Jul 7 21:21:54 omap1-pri multipathd: sda: tur checker reports path is down
Jul 7 21:22:04 omap1-pri multipathd: sdb: tur checker reports path is down
Jul 7 21:22:04 omap1-pri multipathd: sda: tur checker reports path is down
Jul 7 21:22:14 omap1-pri multipathd: sdb: tur checker reports path is down
Jul 7 21:22:14 omap1-pri multipathd: sda: tur checker reports path is up
Jul 7 21:22:14 omap1-pri multipathd: 8:0: reinstated
Jul 7 21:22:14 omap1-pri multipathd: my: remaining active paths: 3
Jul 7 21:22:14 omap1-pri multipathd: sdb: tur checker reports path is up
Jul 7 21:22:14 omap1-pri multipathd: 8:16: reinstated
Jul 7 21:22:14 omap1-pri multipathd: my: remaining active paths: 4
Jul 7 21:22:14 omap1-pri multipathd: dm-2: add map (uevent)
Jul 7 21:22:14 omap1-pri multipathd: dm-2: devmap already registered
Jul 7 21:22:14 omap1-pri multipathd: dm-2: add map (uevent)
Jul 7 21:22:14 omap1-pri multipathd: dm-2: devmap already registered

It just have few IO error right after im plug out the link; it seem difference with the one use /dev/mpath/my (1 2 3). Is it the root cause of my case?

Hi Steve

Here is my fstab:


/dev/VolGroup01/LogVol00 / ext3 defaults 1 1
LABEL=/boot /boot ext3 defaults 1 2
tmpfs /dev/shm tmpfs defaults 0 0
devpts /dev/pts devpts gid=5,mode=620 0 0
sysfs /sys sysfs defaults 0 0
proc /proc proc defaults 0 0
/dev/VolGroup01/LogVol01 swap swap defaults 0 0

Hi

Maybe i see my issue:
In failover successful server:

oracleasm querydisk -d DATA
Disk "DATA" is a valid ASM disk on device /dev/dm-5[253,5]

In failover problem server;

oracleasm querydisk -d DATA
Disk "DATA" is a valid ASM disk on device /dev/sda3[8,3]

ASM use sda so when it lose sda, it still try to send out to sda; i think that s the reason why i see alot of IO error even multipathd dectected link down. ASM mapping trouble.

Yes that is exactly the issue. As noted previously references to the sd* devices are only going to that particular component and NOT to the multipath device. You'd need to change it to use the multipath device.

Also as mentioned in my prior post you should NOT use the dm-* device in your ASM definition as the number can change on reboot. Instead use your my* devices which should not change on boot.

Did you use asmlib and/or udev to setup these devices?

Note: You can run Oracle 11gR2 RAC/GRID on RHEL6 so if this is a new Oracle cluster you're creating you might want to move to RHEL6 instead of RHEL5 as RHEL5 is extremely old. (FYI: RHEL6 allows you to do ASM but does not have asmlib you setup is a tad different so far as udev goes.)

Hi Mensa

I installed Oracle RAC 11gR2 in RedHat 5.7 and i use ASMlib to setup ASM for Oracle RAC

I already setup 2 new server for Oracle RAC and using map ASM disk to /dev/mapper/my (1 2 3). But when i use querydisk command, it still show ASM disk use: dm-*

oracleasm querydisk -d DATA
Disk "DATA" is a valid ASM disk on device /dev/dm-5[253,5]

Is it ok?. I tried reboot server and plug out fiber link one by one and ASM failover successful.

I think after finish all of this, i will try Oracle RAC 12c in RHEL 6; is it better performance for Oracle ?

Best regards,

TrungNQ