LUN Failing and gong to read only mode

bentech4u · 03-01-2012, 10:06 AM

My Server is IBM Blade central HS22.we have one storagetek 6780.Total three Luns are mapped to this server and one of them is failing.other 2 are working perfectly.

after failing the LUN gong to read only mode and after remount it's in read/erite mode.this happened 4 times

Code:

Mar  1 02:07:07 KG-P-LNX-04-FLS kernel: 99 [RAIDarray.mpp]kg-p-6780-stor:0:0:2 mpp status 7
Mar  1 02:07:07 KG-P-LNX-04-FLS kernel: 7 [RAIDarray.mpp]kg-p-6780-stor:0:0 Path Failed
Mar  1 02:07:07 KG-P-LNX-04-FLS kernel: 495 [RAIDarray.mpp]kg-p-6780-stor:0:0:2 Cmnd failed-retry on a new path. vcmnd SN 61085580 pdev H1:C0:T0:L2 0x00/0x00/0x00 0x00070000 mpp_status:7
Mar  1 02:07:47 KG-P-LNX-04-FLS kernel: 99 [RAIDarray.mpp]kg-p-6780-stor:0:1:2 mpp status 7
Mar  1 02:07:47 KG-P-LNX-04-FLS kernel: 7 [RAIDarray.mpp]kg-p-6780-stor:0:1 Path Failed
Mar  1 02:07:47 KG-P-LNX-04-FLS kernel: 495 [RAIDarray.mpp]kg-p-6780-stor:0:1:2 Cmnd failed-retry on a new path. vcmnd SN 61085790 pdev H1:C0:T2:L2 0x00/0x00/0x00 0x00070000 mpp_status:7
Mar  1 02:07:57 KG-P-LNX-04-FLS nmbd[11085]: [2012/03/01 02:07:57, 0] nmbd/nmbd_namequery.c:query_name_response(109) 
Mar  1 02:07:57 KG-P-LNX-04-FLS nmbd[11085]:   query_name_response: Multiple (2) responses received for a query on subnet 10.64.10.129 for name MYGROUP<1d>. 
Mar  1 02:07:57 KG-P-LNX-04-FLS nmbd[11085]:   This response was from IP 10.64.10.130, reporting an IP address of 10.64.10.130. 
Mar  1 02:08:13 KG-P-LNX-04-FLS kernel: 99 [RAIDarray.mpp]kg-p-6780-stor:0:0:2 mpp status 7
Mar  1 02:08:13 KG-P-LNX-04-FLS kernel: 7 [RAIDarray.mpp]kg-p-6780-stor:0:0 Path Failed
Mar  1 02:08:13 KG-P-LNX-04-FLS kernel: 495 [RAIDarray.mpp]kg-p-6780-stor:0:0:2 Cmnd failed-retry on a new path. vcmnd SN 61085945 pdev H1:C0:T0:L2 0x00/0x00/0x00 0x00070000 mpp_status:7
Mar  1 02:08:14 KG-P-LNX-04-FLS kernel: 500 [RAIDarray.mpp]kg-p-6780-stor:1:0:2 Fail to current ctrl 1.. vcmnd SN 61085945 pdev H1:C0:T1:L2 0x05/0x94/0x01 0x08000002 mpp_status:1
Mar  1 02:08:14 KG-P-LNX-04-FLS kernel: 10 [RAIDarray.mpp]kg-p-6780-stor:1 Failover command issued
Mar  1 02:08:15 KG-P-LNX-04-FLS kernel: 801 [RAIDarray.mpp]Failover succeeded to kg-p-6780-stor:1
Mar  1 02:08:25 KG-P-LNX-04-FLS kernel: 99 [RAIDarray.mpp]kg-p-6780-stor:1:0:2 mpp status 7
Mar  1 02:08:25 KG-P-LNX-04-FLS kernel: 7 [RAIDarray.mpp]kg-p-6780-stor:1:0 Path Failed
Mar  1 02:08:25 KG-P-LNX-04-FLS kernel: 495 [RAIDarray.mpp]kg-p-6780-stor:1:0:2 Cmnd failed-retry on a new path. vcmnd SN 61085972 pdev H1:C0:T1:L2 0x00/0x00/0x00 0x00070000 mpp_status:7
Mar  1 02:08:43 KG-P-LNX-04-FLS kernel: 171 [RAIDarray.mpp]kg-p-6780-stor:1:1:2 Retry Count Exhausted: mpp status 7
Mar  1 02:08:43 KG-P-LNX-04-FLS kernel: 492 [RAIDarray.mpp]kg-p-6780-stor:1:1:2 IO FAILURE. vcmnd SN 61086048 pdev H1:C0:T3:L2 0x00/0x00/0x00 0x00070000 mpp_status:7
Mar  1 02:08:43 KG-P-LNX-04-FLS kernel: sd 3:0:0:2: SCSI error: return code = 0x00070000
Mar  1 02:08:43 KG-P-LNX-04-FLS kernel: end_request: I/O error, dev sdd, sector 2644310199
Mar  1 02:08:43 KG-P-LNX-04-FLS kernel: device-mapper: multipath: Failing path 8:48.
Mar  1 02:08:43 KG-P-LNX-04-FLS multipathd: dm-2: add map (uevent) 
Mar  1 02:08:43 KG-P-LNX-04-FLS multipathd: dm-2: devmap already registered 
Mar  1 02:08:43 KG-P-LNX-04-FLS multipathd: 8:48: mark as failed 
Mar  1 02:08:43 KG-P-LNX-04-FLS multipathd: mpath3: remaining active paths: 0 
Mar  1 02:08:48 KG-P-LNX-04-FLS multipathd: sdd: readsector0 checker reports path is up 
Mar  1 02:08:48 KG-P-LNX-04-FLS multipathd: 8:48: reinstated 
Mar  1 02:08:48 KG-P-LNX-04-FLS multipathd: mpath3: remaining active paths: 1 
Mar  1 02:08:48 KG-P-LNX-04-FLS multipathd: dm-2: add map (uevent) 
Mar  1 02:08:48 KG-P-LNX-04-FLS multipathd: dm-2: devmap already registered 
Mar  1 02:10:10 KG-P-LNX-04-FLS kernel: 494 [RAIDarray.mpp]kg-p-6780-stor:1:1:2 Cmnd-failed try alt ctrl 0. vcmnd SN 61086106 pdev H1:C0:T3:L2 0x05/0x94/0x01 0x08000002 mpp_status:1
Mar  1 02:12:57 KG-P-LNX-04-FLS nmbd[11085]: [2012/03/01 02:12:57, 0] nmbd/nmbd_namequery.c:query_name_response(109) 
Mar  1 02:12:57 KG-P-LNX-04-FLS nmbd[11085]:   query_name_response: Multiple (2) responses received for a query on subnet 10.64.10.129 for name MYGROUP<1d>. 
Mar  1 02:12:57 KG-P-LNX-04-FLS nmbd[11085]:   This response was from IP 10.64.10.130, reporting an IP address of 10.64.10.130. 
Mar  1 02:17:26 KG-P-LNX-04-FLS kernel: 99 [RAIDarray.mpp]kg-p-6780-stor:0:0:2 mpp status 7
Mar  1 02:17:26 KG-P-LNX-04-FLS kernel: 7 [RAIDarray.mpp]kg-p-6780-stor:0:0 Path Failed
Mar  1 02:17:26 KG-P-LNX-04-FLS kernel: 495 [RAIDarray.mpp]kg-p-6780-stor:0:0:2 Cmnd failed-retry on a new path. vcmnd SN 61086869 pdev H1:C0:T0:L2 0x00/0x00/0x00 0x00070000 mpp_status:7
Mar  1 02:17:40 KG-P-LNX-04-FLS kernel: 99 [RAIDarray.mpp]kg-p-6780-stor:0:1:2 mpp status 7
Mar  1 02:17:40 KG-P-LNX-04-FLS kernel: 7 [RAIDarray.mpp]kg-p-6780-stor:0:1 Path Failed
Mar  1 02:17:40 KG-P-LNX-04-FLS kernel: 495 [RAIDarray.mpp]kg-p-6780-stor:0:1:2 Cmnd failed-retry on a new path. vcmnd SN 61086925 pdev H1:C0:T2:L2 0x00/0x00/0x00 0x00070000 mpp_status:7
Mar  1 02:17:41 KG-P-LNX-04-FLS kernel: 500 [RAIDarray.mpp]kg-p-6780-stor:1:0:2 Fail to current ctrl 1.. vcmnd SN 61086925 pdev H1:C0:T1:L2 0x05/0x94/0x01 0x08000002 mpp_status:1
Mar  1 02:17:41 KG-P-LNX-04-FLS kernel: 10 [RAIDarray.mpp]kg-p-6780-stor:1 Failover command issued
Mar  1 02:17:42 KG-P-LNX-04-FLS kernel: 801 [RAIDarray.mpp]Failover succeeded to kg-p-6780-stor:1
Mar  1 02:18:05 KG-P-LNX-04-FLS nmbd[11085]: [2012/03/01 02:18:05, 0] nmbd/nmbd_namequery.c:query_name_response(109) 
Mar  1 02:18:05 KG-P-LNX-04-FLS nmbd[11085]:   query_name_response: Multiple (2) responses received for a query on subnet 10.64.10.129 for name MYGROUP<1d>. 
Mar  1 02:18:05 KG-P-LNX-04-FLS nmbd[11085]:   This response was from IP 10.64.10.130, reporting an IP address of 10.64.10.130. 
Mar  1 02:18:09 KG-P-LNX-04-FLS kernel: 99 [RAIDarray.mpp]kg-p-6780-stor:1:0:2 mpp status 7
Mar  1 02:18:09 KG-P-LNX-04-FLS kernel: 7 [RAIDarray.mpp]kg-p-6780-stor:1:0 Path Failed
Mar  1 02:18:09 KG-P-LNX-04-FLS kernel: 495 [RAIDarray.mpp]kg-p-6780-stor:1:0:2 Cmnd failed-retry on a new path. vcmnd SN 61090321 pdev H1:C0:T1:L2 0x00/0x00/0x00 0x00070000 mpp_status:7
Mar  1 02:18:35 KG-P-LNX-04-FLS kernel: 171 [RAIDarray.mpp]kg-p-6780-stor:1:1:2 Retry Count Exhausted: mpp status 7
Mar  1 02:18:35 KG-P-LNX-04-FLS kernel: 492 [RAIDarray.mpp]kg-p-6780-stor:1:1:2 IO FAILURE. vcmnd SN 61090775 pdev H1:C0:T3:L2 0x00/0x00/0x00 0x00070000 mpp_status:7
Mar  1 02:18:35 KG-P-LNX-04-FLS kernel: sd 3:0:0:2: SCSI error: return code = 0x00070000
Mar  1 02:18:35 KG-P-LNX-04-FLS kernel: end_request: I/O error, dev sdd, sector 2645218831
Mar  1 02:18:35 KG-P-LNX-04-FLS kernel: device-mapper: multipath: Failing path 8:48.
Mar  1 02:18:35 KG-P-LNX-04-FLS multipathd: dm-2: add map (uevent) 
Mar  1 02:18:35 KG-P-LNX-04-FLS multipathd: dm-2: devmap already registered 
Mar  1 02:18:35 KG-P-LNX-04-FLS kernel: Buffer I/O error on device dm-3, logical block 7865
Mar  1 02:18:35 KG-P-LNX-04-FLS kernel: lost page write due to I/O error on dm-3
Mar  1 02:18:35 KG-P-LNX-04-FLS kernel: Aborting journal on device dm-3.
Mar  1 02:18:35 KG-P-LNX-04-FLS kernel: Buffer I/O error on device dm-3, logical block 1545
Mar  1 02:18:35 KG-P-LNX-04-FLS kernel: lost page write due to I/O error on dm-3
Mar  1 02:18:35 KG-P-LNX-04-FLS multipathd: 8:48: mark as failed 
Mar  1 02:18:35 KG-P-LNX-04-FLS multipathd: mpath3: remaining active paths: 0 
Mar  1 02:18:40 KG-P-LNX-04-FLS multipathd: sdd: readsector0 checker reports path is up 
Mar  1 02:18:40 KG-P-LNX-04-FLS multipathd: 8:48: reinstated 
Mar  1 02:18:40 KG-P-LNX-04-FLS multipathd: mpath3: remaining active paths: 1 
Mar  1 02:18:40 KG-P-LNX-04-FLS multipathd: dm-2: add map (uevent) 
Mar  1 02:18:40 KG-P-LNX-04-FLS multipathd: dm-2: devmap already registered 
Mar  1 02:18:59 KG-P-LNX-04-FLS kernel: ext3_abort called.
Mar  1 02:18:59 KG-P-LNX-04-FLS kernel: EXT3-fs error (device dm-3): ext3_journal_start_sb: Detected aborted journal
Mar  1 02:18:59 KG-P-LNX-04-FLS kernel: Remounting filesystem read-only

this is from messages file.

Code:

[root@KG-P-LNX-04-FLS mapper]# multipath -d -l
mpath2 (3600a0b80006e2228000007124e6eb575) dm-1 SUN,StorageArray
[size=1.9T][features=0][hwhandler=0][rw]
\_ round-robin 0 [prio=0][active]
 \_ 3:0:0:1 sdc 8:32  [active][undef]
mpath1 (3600a0b80006e23d20000073a4e6eb778) dm-0 SUN,StorageArray
[size=1.9T][features=0][hwhandler=0][rw]
\_ round-robin 0 [prio=0][active]
 \_ 3:0:0:0 sdb 8:16  [active][undef]
mpath3 (3600a0b80006e23d20000073d4e6eb816) dm-2 SUN,StorageArray
[size=1.9T][features=0][hwhandler=0][rw]
\_ round-robin 0 [prio=0][active]
 \_ 3:0:0:2 sdd 8:48  [active][undef]

here it says links are active..How to fix this issue.very urgent and i can't find any solution for this.didn't tried reboot, bcoz some critical applications are running

Thanks
Ben

bentech4u · 03-01-2012, 10:08 AM

before it was working from last 6 months and 2 days back only this issue started.

bentech4u · 03-01-2012, 10:10 AM

More status of Multipath.

Code:

[root@KG-P-LNX-04-FLS mapper]# dmsetup info mpath1p1 
Name:              mpath1p1
State:             ACTIVE
Read Ahead:        256
Tables present:    LIVE
Open count:        1
Event number:      0
Major, minor:      253, 5
Number of targets: 1
UUID: part1-mpath-3600a0b80006e23d20000073a4e6eb778

[root@KG-P-LNX-04-FLS mapper]# dmsetup info mpath2p1 
Name:              mpath2p1
State:             ACTIVE
Read Ahead:        256
Tables present:    LIVE
Open count:        1
Event number:      0
Major, minor:      253, 4
Number of targets: 1
UUID: part1-mpath-3600a0b80006e2228000007124e6eb575

[root@KG-P-LNX-04-FLS mapper]# dmsetup info mpath3p1 
Name:              mpath3p1
State:             ACTIVE
Read Ahead:        256
Tables present:    LIVE
Open count:        1
Event number:      0
Major, minor:      253, 3
Number of targets: 1
UUID: part1-mpath-3600a0b80006e23d20000073d4e6eb816

anomie · 03-01-2012, 10:27 AM

What does the Storagetek report about the LUN's status? Are there any switches connecting you to the storage? If so, what state are their ports in?

bentech4u · 03-01-2012, 11:15 AM

from the storage side i didn't checked..i checked with IBM bladecentral utility.There is no error related no this particular balde server

can we suspect Storage..? if so i will request storage admin to check that..Before checking they are saying that this is the problem with linux side.

anomie · 03-01-2012, 02:22 PM

No speculative suspecting / no pointing fingers. Gather evidence and present your hypothesis.

Before you contact them, we should probably eliminate a filesystem issue first. Unmount your LUN and fsck(8) it.

bentech4u · 03-02-2012, 01:38 AM

k i will try Fsck and let you know the status.

i have one more doubt. what does it mean "[RAIDarray.mpp]kg-p-6780-stor:1:1:2 mpp status 7"

bentech4u · 03-02-2012, 03:04 AM

HI

is there any additional configuration required to perform multipathing..?

at the time of installation i compiled "Sun StorageTek RDAC Multipath Failover Driver for Linux OS" package

after that i did below steps..?

Code:

fdisk /dev/sdb  and create partation

kpartx -a /dev/mapper/mpath0

mkfs.ext3 /dev/mapper/mpath0p1

and finally mounted /dev/mapper/mpath0p1 to /data2

is there anything wrong with these steps.?

anomie · 03-02-2012, 09:13 AM

What did the fsck(8) report?

And what do you mean, "you compiled" that package? I have no idea what distro / version you're using, but on recent RHEL (for instance) DM-Multipath is already supported, and its packages are available via the RHN. There's typically no need to compile drivers or build packages.

bentech4u · 03-02-2012, 09:27 AM

I installed "Sun StorageTek RDAC Multipath Failover Driver" this package downloaded from support.oracle.com.

i got this information from http://docs.oracle.com/cd/E19373-01/.../chapsing.html

my OS is RHEL 5.4

anomie · 03-02-2012, 09:32 AM

I can't help with that third party driver. Carefully read the prerequisites and documentation, and go over your steps again. Of particular interest may be this note:

Quote:

Note - The RDAC driver cannot co-exist with an HBA-level failover driver such as the Emulex, QLogic, or LSI Logic HBA failover drivers.

I use a StorageTek 9985V on RHEL5 with no special drivers. Native DM-Multipath, RHEL-bundled HBA drivers, et al. work perfectly. (I realize you have a different model, so it's likely a very different animal.)

bentech4u · 03-02-2012, 09:42 AM

at that time i am getting lot of I/O errors at the time of OS starting.After installation this driver only got solved..that's why i installed this.also from the Storage team they suggested me to install this driver.

now what to do.? can u please give me the configuration method of native Dm-Multipath configuration.?

is it possible to change this setup to native without data lose.?

please help me.i am not that much expert in this.please

Thanks

anomie · 03-02-2012, 09:55 AM

RHEL systems attached to SANs produce copious I/O error chatter related to LUNs at boot time. That may be perfectly normal.

What I would do is this:

Inform your customer about an outage needed for troubleshooting, if needed.
Run the fsck(8) we already talked about. Don't do anything else until this is done.
If that does not resolve the problem, remove the driver you installed (assuming you can do so non-destructively).
Read and understand the DM-Multipath documentation (http://docs.redhat.com/docs/en-US/Re...ath/index.html), and implement for your environment.

There is no substitute for thoroughly reviewing the documentation. Successfully configuring /etc/multipath.conf depends on it.

I would not install Oracle's "multipath drivers" unless/until I determined RHEL's native support was not sufficient.