The curious case of missing scsi disks..

nkiran · 04-18-2012, 07:46 AM

Hello,
I have a linux server (RHEL6.2) which is connected to a Hitachi storage with 28 disks. There are 4 paths from server to storage.

If i use the "multipath" or "fdisk -l" command to see number of disks; it shows only 26 disks.

However, if i do "lsscsi", I can see all 28 disks (It shows 28 disks * 4 paths = 112 disks).

What could be wrong here? I have rebooted the server as well as done rescanning of the disks using "scsi-rescan"; but no change in result.

Any idea, folks?

MensaWater · 04-18-2012, 08:55 AM

Are all the disks shown by lsscsi showing up as Hitachi OPEN-V?

Do all the disks shown by lsscsi show a device path (/dev/sd...)? If they do have you tried running a for loop on the paths shown by lsscsi to see which if any of these device paths are NOT showing output with fdisk?

Code:

for dev in $(lsscsi |grep "HITACHI  OPEN-V" |awk '{print $NF}')
do fdisk -l $dev
echo "====================================================================="
done

jefro · 04-19-2012, 12:02 AM

Might be able to use the array to help diag it. I am not familiar with that system so I can't say what all if available from the front panel.

nkiran · 04-19-2012, 12:37 AM

MensaWater & Jefro, thank you for your replies.

Yes, all the devices shown by lsscsi show path. Two of these devices don't show output for "fdisk -l".

bhakti~> fdisk -l /dev/sdcp
bhakti~> fdisk -l /dev/sdam

What are things that I should be doing?

MensaWater · 04-19-2012, 07:39 AM

Do you have the Hitachi HORCM software installed on the client? If so run:

ls /dev/sd* |inqraid -CLI -fx -fn >ls_dev_sd_inqraid_CLI_fx_fn.out

Review the out file created. It should have entries like:
DEVICE_FILE PORT SERIAL LDEV CTG H/M/12 SSID R:Group LDEV_NIC_NAME
sda - - - - - - - -
sdb - - - - - - - -
sdc CL2-F 53570 5E00 - s/s/ss 0062 A:00005 MSEXCHANGE_GRT_000
sdd CL2-F 53570 5E01 - s/s/ss 0062 A:00005 MSEXCHANGE_GRT_001
sde CL2-F 53570 5E02 - s/s/ss 0062 A:00005 MSEXCHANGE_GRT_002
sdf CL2-F 53570 5E03 - s/s/ss 0062 A:00005 MSEXCHANGE_GRT_003
sdg CL1-F 53570 5E00 - s/s/ss 0062 A:00005 MSEXCHANGE_GRT_000
sdh CL1-F 53570 5E01 - s/s/ss 0062 A:00005 MSEXCHANGE_GRT_001
sdi CL1-F 53570 5E02 - s/s/ss 0062 A:00005 MSEXCHANGE_GRT_002
sdj CL1-F 53570 5E03 - s/s/ss 0062 A:00005 MSEXCHANGE_GRT_003

The PORT is the port from the Hitachi array and you should see an entry for each port that has paths assigned to this host. The serial number will be the same for all the disks from the same array. The LDEV is the LDEV (CU

EV without the colon) as seen on the Hitachi array. If you have multiple paths to the host from the array you should see the same LDEV on different PORTs on the array. (e.g. in above you see I have the same LDEVs presented on both port CL2-F and CL1-F.)

In the Hitachi disk array web interface you can pull up individual LDEVs and see what ports are assgined to them. From what you said I suspect you're missing port assignments on an LDEV.

Also it might be useful to do a count of just the disks from a single port and compare with other ports you see:
e.g.
grep CL2-F ls_dev_sd_inqraid_CLI_fx_fn.out |wc -l
should equal
grep CL1-F ls_dev_sd_inqraid_CLI_fx_fn.out |wc -l
It may be all your disks are showing up but simply not showing up on all the paths you expect which again would be indicative of a missing PORT assignment on the LDEV in the array itself.

nkiran · 04-19-2012, 09:06 AM

No, I do not have that CLI utility installed. I am using the web interface & I have never used it before.
I found everything on place, all ports & LUNs. I can see 28 LUNs with ports assigned to it.
I have attached couple of screenshots as well.. Just so that you can confirm if I am viewing the correct window.
Thanks again.

Kiran

MensaWater · 04-19-2012, 09:43 AM

Does "cat /proc/scsi/scsi" show the correct number of Hitachi OPEN-V devices?

I'd recommend getting and installing the HORCM software on your Linux system. inqraid will show you things as the server is seeing where as the Hitachi Storage Navigator is only showing you as the array is seeing.

I had an issue a while back where there were missing devices after I added new drives to an existing host. This led me into finding the /dev/sd* devices hadn't been created which I ended up doing a "mknod" to do then I found the sg devices (scsi control) weren't created and also did the "mknod" but that didn't solve the problem. Eventually rebooting the server did solve it and it changed the sd/sg devices associated with disks on reboot so I don't know if simply rebooting it helped or if first doing the mknod then rebooting made it re-examine those major/minor number devices I'd created.

Some of what I do on adding disk to a host with Qlogic HBAs - you might want to examine it to see if any of the information helps you.
WARNING: The resets reset the fibre cards - if you have multipathing enabled or the disks aren't currently in use doing resets on each one at a time shouldn't cause issues but if you are relying on a single path for a mounted filesystem then you probably want to stop the app and unmount the filesystem before doing any resets.
1) Login and become root then cd /sys/class/scsi_host.
2) Run echo “- - -“ > hostX/scan to force the card to rescan the devices presented with hostX being the instance number of the hosts listed that were FC cards. (Note: There are spaces between the dashes after the echo above.)
3) Run "sg_reset -h on /dev/sd of first fibre card to reset it.
4) Wait for first fibre card reset to complete then run "sg_reset -h on /dev/sd of second fibre card to reset it.
5) Run "root/qlogic/ql-dynamic-tgt-lun-disc-2.5/ql-dynamic-tgt-lun-disc.sh" to force Qlogic adpaters to rescan for new devices. (This utility comes from Qlogic.)
6) Run "ls /dev/sd* |inqraid -fx -CLI > ls_dev_sd_inqraid_fx_CLI.20110321_after_ql-dynamic-tgt-lun-disc_sh_script" then check output file to verify the new LDEVs are showing up. Note the device names used (e.g. sddb, sddj).
7) Run sd_block.sh (script we use here):

Code:

#!/bin/bash
#
# Script to get list of sys block devices with VENDOR name and MODEL
cd /sys/block
for each in $(ls -d sd*)
do A=`scsi_id -g -u -s /block/$each`
   B=` scsi_id -g -a -p0x80 -x -s /block/$each | awk '/VENDOR/ {printf "%s ",$0} /MODEL/ {print $0}'`
   echo $each $A $B
done