How to configure GPFS nsd using LVM

vasuhari · 03-27-2010, 08:36 PM

Hi,

I am currently working on configuring GPFS files system under RHEL4 U8. In my setup I am using the following,

1. RHEL4 U8 ES 32-bit.
2. GPFS 3.2.1.
3. Configured GPFS cluster.
4. Configured DM-MP multipathing.
5. Configured LVM with the available SAN luns (from Netapp Storage).
6. I could bring the dm-mp devices under GPFS and I have configured GPFS file systems.

But I want to create nsd (gpfs) with LVM. Is it possible to create gpfsnsd on lvm.

When I tried to run the command,

# ./mmcrnsd -F /gpfs/gpfs.disks -v yes
mmcrnsd: Processing disk vg1/lvol0
mmcrnsd: vg1/lvol0 was not found in /proc/partitions.
mmcrnsd: Failed while processing disk descriptor /dev/vg1/lvol0:server1

rayfordj · 04-17-2010, 11:12 AM

While I have never heard of GPFS before, the first hit on google for mmcrnsd took me to GPFS V3.1 Administration and Programming Reference. Reading over that, particularly regarding the DiskName and Examples section, it leads me to believe that mmcrnsd is expecting a block-device|partition that is directly under /dev/ and found in /proc/partitions.

My initial inclination is to identify what the dm-# for the LV is in /proc/partitions and /dev/ and use that instead of vg1/lvol0 in DescFile (/gpfs/gpfs.disks) and see if that works better for you.

To find the dm-# to use (for example):

Code:

# lvdisplay -m /dev/vg1/lvol0 | grep Block
  Block device           253:5

# grep -e 253[[:space:]]*5 /proc/partitions
 253     5   67108864 dm-5

# ls -l /dev/dm-5
brw-rw---- 1 root disk 253, 5 2010-01-20 08:07 /dev/dm-5

datamove · 02-07-2011, 06:41 AM

Hi, Rayfordj

I bumped into the same problem, but I don't have a these /dev/dm-X entries corresponding to my LVM volumes. If I create soft links in /dev like /dev/dm-3 -> /dev/mapper/myraid-03 then all works fine for me. But these soft links don't survive a reboot of cource. How can I set the system so that /dev/dm-X devices automatically appear and match those in /proc/partitions?

cheers
Artem.

rayfordj · 02-07-2011, 07:08 PM

Quote:

Originally Posted by datamove

Hi, Rayfordj

I bumped into the same problem, but I don't have a these /dev/dm-X entries corresponding to my LVM volumes. If I create soft links in /dev like /dev/dm-3 -> /dev/mapper/myraid-03 then all works fine for me. But these soft links don't survive a reboot of cource. How can I set the system so that /dev/dm-X devices automatically appear and match those in /proc/partitions?

cheers
Artem.

It is possible the default(s) for the version you are using do not interact as expected. You may want to consult lvm.conf for the values of the configured udev parameters (udev_sync and udev_rules).

What OS/release/architecture? What lvm2 and udev package versions are installed?

datamove · 02-08-2011, 02:44 AM

Hi, Rayford,

Thanks for your reply. This is RH5.5 clone (Scientific Linux) x86_64. Udev is Version 095, Release 14.21.el5. LVM2 is Version 2.02.56, Release 8.el5_5.6

udev_sync=1 in the lvm.conf, no udev_rules

cheers
Artem.

datamove · 02-08-2011, 04:14 AM

Hi, Rayford

Thans for hints with udev rules, I went to /etc/udev/rules.d/90-dm.rules and commented out the last option, and now /dev/dm-X nodes are created.

KERNEL=="dm-[0-9]*", ACTION=="add"
#"OPTIONS+="ignore_device"

Although, it's not clear to me why the option was there on the first hand and how this will affect some other parts of the system...

cheers
Artem.