Problems setting up iScsi with Multipath on OEL4.5 (RHEL branded Oracle)

muduck18 · 12-18-2009, 09:55 AM

Can anyone tell me how to configure iScsi multipath on Oracle Enterprise Linux (RHEL 4.5)

I set up a test box to try it out on, and I have had No luck.
installed the RPMs
[root@linuxTest ~]# rpm -qa iscsi* device-mapper*
device-mapper-multipath-0.4.5-31.el4
iscsi-initiator-utils-4.0.3.0-8
device-mapper-1.02.25-2.el4
[root@linuxTest ~]#

enabled multipath in the iscsi.conf file

setup the multipath.conf file as best I could tell.

and I get this
[root@linuxTest ~]# multipath -ll
TESTmultipath (36090a0289089e35a1aa0744bea0e9582)
[size=10 GB][features="1 queue_if_no_path"][hwhandler="0"]
\_ round-robin 0 [prio=1][active]
\_ 5:0:0:0 sdb 8:16 [active][ready]

SANmultipath (36090a01880288c5cc192b4f273d04a17)
[size=50 GB][features="1 queue_if_no_path"][hwhandler="0"]
\_ round-robin 0 [prio=1][active]
\_ 6:0:0:0 sdc 8:32 [active][ready]

[root@linuxTest ~]# mount LABEL=SAN
mount: LABEL=SAN duplicate - not mounted
[root@linuxTest ~]#

What settings do I need to put in my files?
What Am I missing?

This is a similar question
http://www.linuxquestions.org/questi...estion-771315/

But I don't have 'iscsiadm' It does not exist on my system... Should it?

This Should most likely be in the 'newb' section because I am new, But the question is somewhat advanced and specific to server related discussion.

I am new to Linux, my company sent me to a class and then loaded oracle on Linux and told me to support it...I am learning but not fast enough.

about a week ago our iScsi mounted drive disappeared from the system. I had setup a network bond as an attempt for network redundancy for the iscsi connections. The Bond however does not seem to overcome network cable failure.
Yes thats right, we had a network cable 'suddenly go bad' and the production database server went down hard and took out the ext3 file system it was using. We did a recovery on the system and its backup and running on a new iscsi Lun. yada yada...

Well I opened a case with Oracle Linux support but an ex-manager let the support csi lapse and their sales team apparently doesn't want my companies money. And the support engineer seems to be very hesitant to help until the sales team takes the money.

cardy · 01-06-2010, 02:02 PM

Looking at the output from the multipath command you have 2 scsi disks sda and sdb.

sda is 10GB in size
sdb is 50GB in size

These disks are not multipathed at all. I would suggest that you first look at the output of the command

Quote:

dmesg

and the log file

Quote:

/var/log/messages

You should especially be interested in what SCSI disks are detected during boot and the loading of the iscsi initiator which should show what disks have been detected from the iscsi server.

As for the bonding driver, this needs careful setup to define the correct parameters for both your server and network setup. Depending on what switches you have connected the bonding driver can work in multiple modes and if you have the wrong bonding driver mode selected then the driver could fail on hardware failure.

This is hard to diagnose without knowing more about your setup both server and network wise.

If you can provide relevant information from dmesg and /var/log/messages it should be possible to identify where your having problems.

Regards

Lee

muduck18 · 01-07-2010, 09:45 AM

This is part of the dmesg that has scsi stuff in it.

Quote:

mptbase: Initiating ioc0 bringup
ioc0: 53C1030: Capabilities={Initiator}
scsi0 : ioc0: LSI53C1030, FwRev=00000000h, Ports=1, MaxQ=128, IRQ=169
Vendor: VMware Model: Virtual disk Rev: 1.0
Type: Direct-Access ANSI SCSI revision: 02
SCSI device sda: 16777216 512-byte hdwr sectors (8590 MB)
sda: cache data unavailable
sda: assuming drive cache: write through
SCSI device sda: 16777216 512-byte hdwr sectors (8590 MB)
sda: cache data unavailable
sda: assuming drive cache: write through
sda: sda1 sda2
Attached scsi disk sda at scsi0, channel 0, id 0, lun 0
Fusion MPT SAS Host driver 3.12.19.00rh
libata version 2.00 loaded.
device-mapper: 4.5.5-ioctl (2006-12-01) initialised: dm-devel@redhat.com
EXT3-fs: INFO: recovery required on readonly filesystem.
EXT3-fs: write access will be enabled during recovery.
kjournald starting. Commit interval 5 seconds
EXT3-fs: dm-0: orphan cleanup on readonly fs
ext3_orphan_cleanup: deleting unreferenced inode 459081
EXT3-fs: dm-0: 1 orphan inode deleted
EXT3-fs: recovery complete.
EXT3-fs: mounted filesystem with ordered data mode.
security: 3 users, 4 roles, 356 types, 26 bools
security: 55 classes, 22665 rules
SELinux: Completing initialization.
SELinux: Setting up existing superblocks.
SELinux: initialized (dev dm-0, type ext3), uses xattr
SELinux: initialized (dev tmpfs, type tmpfs), uses transition SIDs
SELinux: initialized (dev selinuxfs, type selinuxfs), uses genfs_contexts
SELinux: initialized (dev mqueue, type mqueue), not configured for labeling
SELinux: initialized (dev hugetlbfs, type hugetlbfs), uses genfs_contexts
SELinux: initialized (dev devpts, type devpts), uses transition SIDs
SELinux: initialized (dev eventpollfs, type eventpollfs), uses genfs_contexts
SELinux: initialized (dev pipefs, type pipefs), uses task SIDs
SELinux: initialized (dev tmpfs, type tmpfs), uses transition SIDs
SELinux: initialized (dev futexfs, type futexfs), uses genfs_contexts
SELinux: initialized (dev sockfs, type sockfs), uses task SIDs
SELinux: initialized (dev proc, type proc), uses genfs_contexts
SELinux: initialized (dev bdev, type bdev), uses genfs_contexts
SELinux: initialized (dev rootfs, type rootfs), uses genfs_contexts
SELinux: initialized (dev sysfs, type sysfs), uses genfs_contexts
SELinux: initialized (dev usbfs, type usbfs), uses genfs_contexts
inserting floppy driver for 2.6.9-78.0.1.ELsmp
Floppy drive(s): fd0 is 1.44M
FDC 0 is a post-1991 82077
VMware vmxnet virtual NIC driver
ACPI: PCI Interrupt 0000:00:11.0[A] -> GSI 18 (level, low) -> IRQ 177
Found vmxnet/PCI at 0x1424, irq 177.
features: ipCsum zeroCopy partialHeaderCopy
divert: allocating divert_blk for eth0
ACPI: PCI Interrupt 0000:00:12.0[A] -> GSI 19 (level, low) -> IRQ 185
Found vmxnet/PCI at 0x14a4, irq 185.
features: ipCsum zeroCopy partialHeaderCopy
divert: allocating divert_blk for eth1
pcnet32.c:v1.31 29.04.2005 tsbogend@alpha.franken.de
md: Autodetecting RAID arrays.
md: autorun ...
md: ... autorun DONE.
SELinux: initialized (dev ramfs, type ramfs), uses genfs_contexts
NET: Registered protocol family 10
Disabled Privacy Extensions on device c0353520(lo)
IPv6 over IPv4 tunneling driver
divert: not allocating divert_blk for non-ethernet device sit0
ACPI: AC Adapter [ACAD] (on-line)
ACPI: Power Button (FF) [PWRF]
EXT3 FS on dm-0, internal journal
device-mapper: dm-multipath version 1.0.5 loaded
kjournald starting. Commit interval 5 seconds
EXT3 FS on sda1, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
SELinux: initialized (dev sda1, type ext3), uses xattr
SELinux: initialized (dev tmpfs, type tmpfs), uses transition SIDs
VMware hgfs: HGFS is disabled in the host
Adding 2097144k swap on /dev/VolGroup00/LogVol01. Priority:-1 extents:1
SELinux: initialized (dev binfmt_misc, type binfmt_misc), uses genfs_contexts
IA-32 Microcode Update Driver: v1.14 <tigran@veritas.com>
microcode: No new microdata for cpu 0
IA-32 Microcode Update Driver v1.14 unregistered
parport0: PC-style at 0x378 [PCSPP,TRISTATE]
ip_tables: (C) 2000-2002 Netfilter core team
ip_tables: (C) 2000-2002 Netfilter core team
ip_tables: (C) 2000-2002 Netfilter core team
eth0: no IPv6 routers present
iscsi-sfnet: Loading iscsi_sfnet version 4:0.1.11-3
iscsi-sfnet: Control device major number 254
eth1: no IPv6 routers present
iscsi-sfnet:host1: Login failed to authenticate with target iqn.2001-05.com.equallogic:0-8a0906-5 ae389902-82950eea4b74a01a-test-linux
iscsi-sfnet:host2: Login failed to authenticate with target iqn.2001-05.com.equallogic:0-8a0906-5 c8c28801-174ad073f2b492c1-linux-onetouch
iscsi-sfnet:host2: Session established
scsi2 : SFNet iSCSI driver
iscsi-sfnet:host1: Session established
scsi1 : SFNet iSCSI driver
Vendor: EQLOGIC Model: 100E-00 Rev: 4.2
Type: Direct-Access ANSI SCSI revision: 05
Vendor: EQLOGIC Model: 100E-00 Rev: 4.2
Type: Direct-Access ANSI SCSI revision: 05
SCSI device sdb: 20981760 512-byte hdwr sectors (10743 MB)
SCSI device sdb: drive cache: write through
SCSI device sdb: 20981760 512-byte hdwr sectors (10743 MB)
SCSI device sdb: drive cache: write through
sdb: sdb1
Attached scsi disk sdb at scsi1, channel 0, id 0, lun 0
SCSI device sdc: 104878080 512-byte hdwr sectors (53698 MB)
SCSI device sdc: drive cache: write through
SCSI device sdc: 104878080 512-byte hdwr sectors (53698 MB)
SCSI device sdc: drive cache: write through
sdc: sdc1
Attached scsi disk sdc at scsi2, channel 0, id 0, lun 0
device-mapper: dm-round-robin version 1.0.0 loaded
SELinux: initialized (dev rpc_pipefs, type rpc_pipefs), uses genfs_contexts
VMware memory control driver initialized
vmmemctl: started kernel thread pid=5219

muduck18 · 01-07-2010, 09:53 AM

I am seeing a couple errors in this one, One I didn't see before the one about the bad value in the iscsi.conf
I thought it was supposed to be multipath=portalgroup

Quote:

Originally Posted by messages

Jan 7 10:36:54 linux-onetouch iscsi: Loading iscsi driver: succeeded
...

Jan 7 10:37:00 linux-onetouch iscsid[4684]: version 4:0.1.11-7 variant (14-Apr-2008)
Jan 7 10:37:00 linux-onetouch iscsi: iscsid startup succeeded
Jan 7 10:37:00 linux-onetouch iscsid[4703]: error on line 86 of /etc/iscsi.conf, ignoring unrecognized line Multipath=portalgroup
Jan 7 10:37:00 linux-onetouch iscsid[4706]: Connected to Discovery Address 172.20.35.1
Jan 7 10:37:00 linux-onetouch kernel: iscsi-sfnet:host1: Login failed to authenticate with target iqn.2001-05.com.equallogic:0-8a0906-5ae389902-82950eea4b74a01a-test-linux
Jan 7 10:37:00 linux-onetouch kernel: iscsi-sfnet:host2: Login failed to authenticate with target iqn.2001-05.com.equallogic:0-8a0906-5c8c28801-174ad073f2b492c1-linux-onetouch
Jan 7 10:37:06 linux-onetouch kernel: iscsi-sfnet:host2: Session established
Jan 7 10:37:06 linux-onetouch kernel: scsi2 : SFNet iSCSI driver
Jan 7 10:37:06 linux-onetouch kernel: iscsi-sfnet:host1: Session established
Jan 7 10:37:06 linux-onetouch kernel: scsi1 : SFNet iSCSI driver
Jan 7 10:37:06 linux-onetouch kernel: Vendor: EQLOGIC Model: 100E-00 Rev: 4.2
Jan 7 10:37:06 linux-onetouch kernel: Type: Direct-Access ANSI SCSI revision: 05
Jan 7 10:37:06 linux-onetouch kernel: Vendor: EQLOGIC Model: 100E-00 Rev: 4.2
Jan 7 10:37:06 linux-onetouch kernel: Type: Direct-Access ANSI SCSI revision: 05
Jan 7 10:37:06 linux-onetouch kernel: SCSI device sdb: 20981760 512-byte hdwr sectors (10743 MB)
Jan 7 10:37:06 linux-onetouch kernel: SCSI device sdb: drive cache: write through
Jan 7 10:37:06 linux-onetouch kernel: SCSI device sdb: 20981760 512-byte hdwr sectors (10743 MB)
Jan 7 10:37:06 linux-onetouch kernel: SCSI device sdb: drive cache: write through
Jan 7 10:37:06 linux-onetouch kernel: sdb: sdb1
Jan 7 10:37:06 linux-onetouch kernel: Attached scsi disk sdb at scsi1, channel 0, id 0, lun 0
Jan 7 10:37:07 linux-onetouch kernel: SCSI device sdc: 104878080 512-byte hdwr sectors (53698 MB)
Jan 7 10:37:07 linux-onetouch scsi.agent[4759]: disk at /devices/platform/host1/target1:0:0/1:0:0:0
Jan 7 10:37:07 linux-onetouch kernel: SCSI device sdc: drive cache: write through
Jan 7 10:37:07 linux-onetouch kernel: SCSI device sdc: 104878080 512-byte hdwr sectors (53698 MB)
Jan 7 10:37:07 linux-onetouch kernel: SCSI device sdc: drive cache: write through
Jan 7 10:37:07 linux-onetouch kernel: sdc: sdc1
Jan 7 10:37:07 linux-onetouch kernel: Attached scsi disk sdc at scsi2, channel 0, id 0, lun 0
Jan 7 10:37:07 linux-onetouch scsi.agent[4808]: disk at /devices/platform/host2/target2:0:0/2:0:0:0
Jan 7 10:37:13 linux-onetouch kernel: device-mapper: dm-round-robin version 1.0.0 loaded
Jan 7 10:37:30 linux-onetouch multipathd: multipathd startup succeeded
Jan 7 10:37:30 linux-onetouch portmap: portmap startup succeeded
Jan 7 10:37:30 linux-onetouch rpc.statd[5136]: Version 1.0.6 Starting
.....

Jan 7 10:37:30 linux-onetouch mount: Error: cannot mount filesystem: Protocol error
Jan 7 10:37:30 linux-onetouch mount: mount: LABEL=SAN duplicate - not mounted
Jan 7 10:37:30 linux-onetouch mount:
Jan 7 10:37:30 linux-onetouch netfs: Mounting other filesystems: failed
Jan 7 10:37:30 linux-onetouch kernel: i2c /dev entries driver
Jan 7 10:37:30 linux-onetouch rc: Starting lm_sensors: succeeded
Jan 7 10:37:30 linux-onetouch autofs: automount startup succeeded
Jan 7 10:37:30 linux-onetouch smartd[5362]: smartd version 5.33 [i386-redhat-linux-gnu] Copyright (C) 2002-4 Bruce Allen
Jan 7 10:37:30 linux-onetouch smartd[5362]: Home page is http://smartmontools.sourceforge.net/
Jan 7 10:37:30 linux-onetouch smartd[5362]: Opened configuration file /etc/smartd.conf
Jan 7 10:37:31 linux-onetouch smartd[5362]: Configuration file /etc/smartd.conf parsed.
Jan 7 10:37:31 linux-onetouch smartd[5362]: Device: /dev/sda, opened
Jan 7 10:37:31 linux-onetouch smartd[5362]: Device: /dev/sda, Bad IEC (SMART) mode page, err=-5, skip device
Jan 7 10:37:31 linux-onetouch smartd[5362]: Unable to register SCSI device /dev/sda at line 30 of file /etc/smartd.conf
Jan 7 10:37:31 linux-onetouch smartd[5362]: Unable to register device /dev/sda (no Directive -d removable). Exiting.

cardy · 01-08-2010, 04:55 AM

Looking at the above it shows that your using VMWare, are you trying to setup the multipath for the VMware Server (ESX etc....) or for a Virtual Machine running on top of a VMWare server.

Normally you would have 2 different networks and tell the client to login to the same SAN via 2 different network interfaces. When the iSCSI initiator scans for available disks it identifies each disk available, disks that have the same uniq reference that are available via 2 different routes are then able to be multipathed by the linux OS.

Can you explain the above setup a little more.

muduck18 · 01-08-2010, 12:40 PM

Yes you are right.
I can see how you would be confused.

I have set up a Linux VM on top of the ESX server. This VM is a test to mimic the production environment so that I can get the multi-path working before turning it loose on our ERP system.

The VM has two network interfaces independently connected to a SAN (virtual switch that connects to an ISCSI network and ISCSI SAN)

Being a windows administrator, If this was a windows machine, I would have been done. because i would go in and specify that I want to connect over NIC0, establish the connection, then open another connection over NIC1. the paths could be set to round robin, or fail over and such.

For the life of me, I can't figure out HOW to set it up so that it will recognize the two paths.
Or even how to figure out how to tell if it does recognize them.

I feel like I am getting Linux dumber and dumber every time I try.

Thank you so much for any help you can offer.