LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices


Reply
  Search this Thread
Old 03-09-2011, 06:09 PM   #1
zenfridge
LQ Newbie
 
Registered: Mar 2011
Posts: 3

Rep: Reputation: 0
RHEL 6 w/LVM volume duplication breaks multipathing


Greetings.

We're trying to take an existing system running multipathed SAN boot disks, duplicate that boot disk to another system volume, and boot a second system up with all the same parts running like the first system (think: cloning the system). However, multipathing is broke on the second system (I'm sure I'm forgetting something stupid here after the cloning). Details:


We have a freshly installed RHEL6 on IBM PPC. This is a boot off of a SAN volume with two paths. Installation went great - multipathing was auto-detected and used underneath the usual LVMs for the boot volumes. Looks and works great:

Code:
[root@goldimage dev]# df -k
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/vg_osbase-lv_root
                      22194044   6917404  14149216  33% /
tmpfs                  3119680         0   3119680   0% /dev/shm
/dev/mapper/mpathap2    495844     43238    427006  10% /boot
/dev/mapper/vg_osbase-lv_var
                      16513960    535468  15139632   4% /var
Code:
[root@goldimage dev]# multipath -ll
mpatha (36005076304ffc7740000000000001023) dm-0 AIX,VDASD
size=50G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
  |- 0:0:1:0 sda 8:0  active ready running
  `- 1:0:1:0 sdc 8:32 active ready running
We shut that system down, use FlashCopy on our DS8100 SAN system [but I think you can insert any other bit level copy method here]. The second system boots surprisingly well (after fixing network/hostname issues) considering the SAN volume WWN has changed - probably because the boot volumes are LVM. It booted up, and then we updated multipath.conf with the new WWN and rebooted again. However, at this point it still refuses to start multipathing. According to dmseg, it does start, but then hits ioctl errors with device mapper:

Code:
device-mapper: multipath: version 1.1.1 loaded
...
device-mapper: multipath round-robin: version 1.0.0 loaded
device-mapper: table: 253:2: multipath: error getting device
device-mapper: ioctl: error adding target to table
device-mapper: table: 253:2: multipath: error getting device
device-mapper: ioctl: error adding target to table
device-mapper: table: 253:2: multipath: error getting device
device-mapper: ioctl: error adding target to table
device-mapper: table: 253:2: multipath: error getting device
device-mapper: ioctl: error adding target to table
...
device-mapper: table: 253:3: multipath: error getting device
device-mapper: ioctl: error adding target to table
device-mapper: table: 253:3: multipath: error getting device
device-mapper: ioctl: error adding target to table
device-mapper: table: 253:3: multipath: error getting device
device-mapper: ioctl: error adding target to table
device-mapper: table: 253:3: multipath: error getting device
device-mapper: ioctl: error adding target to table

[root@goldimage2 disk]# multipathd -k
multipathd> show paths
hcil    dev dev_t pri dm_st chk_st dev_st  next_check
0:0:1:0 sda 8:0   1   undef ready  running orphan
1:0:1:0 sdb 8:16  1   undef ready  running orphan
Suggestions? I would guess there's other cruft from the imaging that needs to get removed/changed, e.g. with device-mapper, but don't know what that would be... Thanks for any help you can provide!! If you'd like other details/output, please let me know!

Last edited by zenfridge; 03-09-2011 at 06:14 PM. Reason: clarity fixes
 
Old 03-11-2011, 07:31 PM   #2
zenfridge
LQ Newbie
 
Registered: Mar 2011
Posts: 3

Original Poster
Rep: Reputation: 0
More details... I'm doing a compare of the two systems (original, copy) dmesg output. In both cases, I do see the same start of dracut, device-mapper and device-mapper multipath:

Code:
dracut: dracut-004-32.el6
dracut: rd_NO_LUKS: removing cryptoluks activation
device-mapper: uevent: version 1.0.3
device-mapper: ioctl: 4.17.0-ioctl (2010-03-05) initialised: dm-devel@redhat.com
udev: starting version 147
device-mapper: multipath: version 1.1.1 loaded
This happens during device probe and startup pretty early on. It then looks like it loads ibmvscsi drivers and probes for disks. That's where most probably a key difference is: It looks like the original system starts multipath round-robin with device-mapper right after disk descovery but BEFORE the LVM disks (e.g. root) are mounted:

Code:
....
sd 1:0:1:0: [sdc] Attached SCSI disk
device-mapper: multipath round-robin: version 1.0.0 loaded
dracut: Scanning devices dm-3 sda3 sdc3  for LVM logical volumes vg_osbase/lv_root vg_osbase/lv_swap
dracut: inactive '/dev/vg_osbase/lv_root' [21.50 GiB] inherit
dracut: inactive '/dev/vg_osbase/lv_swap' [12.00 GiB] inherit
dracut: inactive '/dev/vg_osbase/lv_var' [16.00 GiB] inherit
EXT4-fs (dm-4): mounted filesystem with ordered data mode
dracut: Mounted root filesystem /dev/mapper/vg_osbase-lv_root
....
The duplicated system doesn't start that multipath round-robin at the same spot... it happens later, LVM is already up, etc., and then I get ioctl errors...

Code:
sd 1:0:1:0: [sdb] Attached SCSI disk
dracut: Scanning devices sda3 sdb3  for LVM logical volumes vg_osbase/lv_root vg_osbase/lv_swap
dracut: Found duplicate PV Qm9BcyONlU6AK9104ChsKNfKM4Ngjq5c: using /dev/sdb3 not /dev/sda3
dracut: inactive '/dev/vg_osbase/lv_root' [21.50 GiB] inherit
dracut: inactive '/dev/vg_osbase/lv_swap' [12.00 GiB] inherit
dracut: inactive '/dev/vg_osbase/lv_var' [16.00 GiB] inherit
dracut: Found duplicate PV Qm9BcyONlU6AK9104ChsKNfKM4Ngjq5c: using /dev/sdb3 not /dev/sda3
EXT4-fs (dm-0): mounted filesystem with ordered data mode
dracut: Mounted root filesystem /dev/mapper/vg_osbase-lv_root
...
dracut: Switching root
...
udev: starting version 147
drivers/net/ibmveth.c: ibmveth: IBM i/pSeries Virtual Ethernet Driver 1.03
vio_register_driver: driver ibmveth registering
sd 0:0:1:0: Attached scsi generic sg0 type 0
sd 1:0:1:0: Attached scsi generic sg1 type 0
device-mapper: multipath round-robin: version 1.0.0 loaded
device-mapper: table: 253:2: multipath: error getting device
device-mapper: ioctl: error adding target to table
...
device-mapper: table: 253:3: multipath: error getting device
device-mapper: ioctl: error adding target to table
...

Again, can provide more details as requested, but remember this is a block level copy of a system where multipath was working fine...

Thanks for any help!
 
Old 03-14-2011, 05:47 PM   #3
zenfridge
LQ Newbie
 
Registered: Mar 2011
Posts: 3

Original Poster
Rep: Reputation: 0
Ok, I seem to have solved this, albeit in a round about way. Details:

It looks like the initramfs file still had old disk path info (specifically the WWID in /etc/multipath.conf). I am not sure why my mkinitrd attempt didn't update that file, but it didn't seem to make any difference when I tried last time. Unfortunately, I cannot retrace that attempt: I then tried dracut for the first time and broke the whole box; I had to copy my original disk again, and so lost the mkinitrd I had originally attempted. I will try it again and see what all mkinitrd didn't update. However, I found a solution by just doing things the hard way:

I unpacked the initramfs, updated the etc/multipath.conf inside, and then repackaged the file:

Code:
# get an unpacked copy of the initrd:
mkdir /tmp/newinitrd-files
cp -p /boot/initramfs-2.6.32-71.el6.ppc64.img /tmp/newinitrd-files/initramfs-2.6.32-71.el6.ppc64.img.gz
cd /tmp/newinitrd-files
gunzip initramfs-2.6.32-71.el6.ppc64.img.gz
cpio -id <initramfs-2.6.32-71.el6.ppc64.img
rm initramfs-2.6.32-71.el6.ppc64.img

# change wwid's in path def and blacklist:
vi etc/multipath.conf

# repack this new initrd:
find . |cpio --create --format='newc' > ../newinitrd
cd ..
gzip newinitrd
mv newinitrd.gz /boot/newramfs-2.6.32-71.el6.ppc64.img

# (add a stanza to yaboot.conf (or grub, etc.) for the new ramfs, keeping existing one as a backup)
# (reboot and test)

So, problem was solved in a roundabout way but atleast I can move on, and I'll figure out how come my mkinitrd attempt didn't work correctly... Thanks to all who at least looked at this!
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
LVM Mount Physical Volume/Logical Volume without a working Volume Group mpivintis Linux - Newbie 10 01-11-2014 08:02 AM
Moving /home to another volume breaks ext3 Trash on new volume uncleboarder Linux - General 3 10-01-2010 11:59 PM
How to change the LVM logical volume owner in RHEL 5.3 srithi Linux - Newbie 3 12-31-2009 01:34 PM
Luks/Lvm after dd duplication brodo Slackware 10 03-12-2009 04:56 PM
Fedora LVM volume group & Physical Volume resize problem gabeyg Fedora 1 05-14-2008 12:26 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Software

All times are GMT -5. The time now is 11:35 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration