Hello, I have a filer that was running Fedora22 until recently. It was serving a ZFS array in two LUNS over fibre channel to two initiators, a CentOS 7 VM (with hardware passthrough of a pci qla hba) and an ESXi 6.0 hypervisor. No issues with this configuration, all machines were reliably able to see the point to point connections. Using 4GB cards manually configured to default to 4GB, no issues with that configuration or speed negotiation.
Storage is not my area, and I barely cobbled this together the first time. Now, I've made a mistake: I changed multiple things at once, because I was sure that _THIS TIME_ I knew all the pieces would work and changing >1 thing at time would be fine. I am prepared to learn from that mistake until my next sure-thing.
I upgraded to Fedora 29, reinstalled targetcli and drivers, configured my ZFS array with a new disk (6x2TB in Raidz2) and then created my backstores/targets and tested everything. No luck. The kernel on the filer can see the fibre when I detach/reattach, but also says an error that I have found precious few references to online:
Quote:
[root@filer ~]# journalctl -fk
-- Logs begin at Fri 2019-03-01 17:09:25 EST. --
Apr 01 17:58:16 filer.ipa.example.com kernel: RTL8211E Gigabit Ethernet r8169-300:00: attached PHY driver [RTL8211E Gigabit Ethernet] (mii_bus hy_addr=r8169-300:00, irq=IGNORE)
Apr 01 17:58:17 filer.ipa.example.com kernel: Rounding down aligned max_sectors from 4294967295 to 4294967288
Apr 01 17:58:17 filer.ipa.example.com kernel: ignoring deprecated emulate_dpo attribute
Apr 01 17:58:17 filer.ipa.example.com kernel: ignoring deprecated emulate_fua_read attribute
Apr 01 17:58:17 filer.ipa.example.com kernel: qla2xxx [0000:01:00.1]-00af:11: Performing ISP error recovery - ha=0000000041933d7a.
Apr 01 17:58:18 filer.ipa.example.com kernel: qla2xxx [0000:01:00.1]-500a:11: LOOP UP detected (4 Gbps).
Apr 01 17:58:18 filer.ipa.example.com kernel: qla2xxx [0000:01:00.0]-00af:10: Performing ISP error recovery - ha=00000000c7164f01.
Apr 01 17:58:19 filer.ipa.example.com kernel: qla2xxx [0000:01:00.0]-500a:10: LOOP UP detected (4 Gbps).
Apr 01 17:58:20 filer.ipa.example.com kernel: qla2xxx [0000:01:00.0]-d034:10: qla24xx_do_nack_work create sess success 00000000151c0013
Apr 01 17:58:20 filer.ipa.example.com kernel: r8169 0000:03:00.0 enp3s0: Link is Up - 1Gbps/Full - flow control off
Apr 01 18:01:48 filer.ipa.example.com kernel: qla2xxx [0000:01:00.0]-500b:10: LOOP DOWN detected (2 5 0 0).
Apr 01 18:01:52 filer.ipa.example.com kernel: qla2xxx [0000:01:00.0]-500a:10: LOOP UP detected (4 Gbps).
|
Things I have confirmed (maybe Fedora 29 slipped something past me and it's painfully obvious?):
module should be loaded -
Quote:
[root@filer ~]# lsmod | grep -i qla
tcm_qla2xxx 45056 13
target_core_mod 409600 26 target_core_file,target_core_iblock,iscsi_target_mod,tcm_qla2xxx,target_core_pscsi,target_core_user
qla2xxx 864256 1 tcm_qla2xxx
nvme_fc 45056 1 qla2xxx
scsi_transport_fc 69632 2 qla2xxx,tcm_qla2xxx
|
target and backstores were created (I've added/removed/added ACLs a few times, no luck) -
Quote:
[root@filer ~]# targetcli
targetcli shell version 2.1.fb48
Copyright 2011-2013 by Datera, Inc and others.
For help on commands, type 'help'.
/> ls
o- / .................................................................................................... ...... [...]
o- backstores ............................................................................................... [...]
| o- block ................................................................................... [Storage Objects: 2]
| | o- ESXI_storage ................................................... [/dev/zd16 (500.0GiB) write-thru activated]
| | | o- alua .................................................................................... [ALUA Groups: 1]
| | | o- default_tg_pt_gp ........................................................ [ALUA state: Active/optimized]
| | o- Media_storage ..................................................... [/dev/zd0 (6.0TiB) write-thru activated]
| | o- alua .................................................................................... [ALUA Groups: 1]
| | o- default_tg_pt_gp ........................................................ [ALUA state: Active/optimized]
| o- fileio .................................................................................. [Storage Objects: 0]
| o- pscsi ................................................................................... [Storage Objects: 0]
| o- ramdisk ................................................................................. [Storage Objects: 0]
o- iscsi ............................................................................................. [Targets: 0]
o- loopback .......................................................................................... [Targets: 0]
o- qla2xxx ........................................................................................... [Targets: 2]
| o- naa.2100001b32813e3e .............................................................................. [gen-acls]
| | o- acls ............................................................................................. [ACLs: 0]
| | o- luns ............................................................................................. [LUNs: 1]
| | o- lun0 ................................................. [block/ESXI_storage (/dev/zd16) (default_tg_pt_gp)]
| o- naa.2101001b32a13e3e .............................................................................. [gen-acls]
| o- acls ............................................................................................. [ACLs: 3]
| | o- naa.2000001b32818c63 .................................................................... [Mapped LUNs: 1]
| | | o- mapped_lun0 ............................................................ [lun0 block/Media_storage (rw)]
| | o- naa.2001001b32a18c63 .................................................................... [Mapped LUNs: 1]
| | | o- mapped_lun0 ............................................................ [lun0 block/Media_storage (rw)]
| | o- naa.2100001b328f1c23 .................................................................... [Mapped LUNs: 1]
| | o- mapped_lun0 ............................................................ [lun0 block/Media_storage (rw)]
| o- luns ............................................................................................. [LUNs: 1]
| o- lun0 ................................................. [block/Media_storage (/dev/zd0) (default_tg_pt_gp)]
o- vhost ............................................................................................. [Targets: 0]
/>
|
The target system (CentOS 7) can't see any exported lun. Nothing has changed on either initiator, they both report the fibre card being present and cheerfully find nothing to mount on the other end of the connection.
Filer is also not in initiator mode:
Quote:
[root@filer ~]# cat /etc/modprobe.d/qla2xxx.conf
options qla2xxx qlini_mode="disabled"
|
My primary initiator (I don't care much about ESXi not seeing things for now) -
Quote:
[root@initiator ~]# lspci | grep -i fibre
13:00.0 Fibre Channel: QLogic Corp. ISP2432-based 4Gb Fibre Channel to PCI Express HBA (rev 03)
|
Quote:
[root@initiator ~]# lsmod | grep qla
qla2xxx 729354 0
scsi_transport_fc 64007 1 qla2xxx
|
Quote:
[root@initiator ~]# rescan-scsi-bus.sh
which: no multipath in (/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin)
Scanning SCSI subsystem for new devices
Scanning host 0 for SCSI target IDs 0 1 2 3 4 5 6 7, all LUNs
Scanning for device 0 0 0 0 ...
OLD: Host: scsi0 Channel: 00 Id: 00 Lun: 00
Vendor: VMware Model: Virtual disk Rev: 1.0
Type: Direct-Access ANSI SCSI revision: 02
Scanning host 1 for SCSI target IDs 0 1 2 3 4 5 6 7, all LUNs
Scanning host 2 for SCSI target IDs 0 1 2 3 4 5 6 7, all LUNs
Scanning host 3 for SCSI target IDs 0 1 2 3 4 5 6 7, all LUNs
Scanning for device 3 0 0 0 ...
OLD: Host: scsi3 Channel: 00 Id: 00 Lun: 00
Vendor: ATA Model: TOSHIBA MK4055GS Rev: 1M
Type: Direct-Access ANSI SCSI revision: 05
Scanning host 4 for SCSI target IDs 0 1 2 3 4 5 6 7, all LUNs
Scanning host 5 for SCSI target IDs 0 1 2 3 4 5 6 7, all LUNs
Scanning for device 5 0 0 0 ...
OLD: Host: scsi5 Channel: 00 Id: 00 Lun: 00
Vendor: ATA Model: ST31500541AS Rev: CC32
Type: Direct-Access ANSI SCSI revision: 05
Scanning host 6 for SCSI target IDs 0 1 2 3 4 5 6 7, all LUNs
Scanning host 7 for all SCSI target IDs, all LUNs
0 new or changed device(s) found.
0 remapped or resized device(s) found.
0 device(s) removed.
|
So, before I go down the rabbit hole of rolling back versions/operating systems etc. I would love for someone to point out that I've missed a super obvious issue.
As always, thanks in advance LQ!
Cheers,
JoseCuervo
