iscsi target

ilesterg · 03-18-2017, 04:32 AM

Hi!

This question goes for Debian based and RHEL based distros, but I believe it will apply to other distros as well.

I'm working on a Virtulbox environment, and I have been trying to set up iscsi targets to mimic a "SAN" and have been successful so far. On a variety of targets, I was able to format the disks using a variety of combinations: LVM mirrored, striped, different types of filesystems etc.

This is how devices are represented in my target's configuration:

Code:

Target iqn.2017-02.net.lcldomain:lun68_1
  Lun 0 Path=/dev/sdb,Type=blockio
  Alias LUN0
Target iqn.2017-02.net.lcldomain:lun73_1
  Lun 1 Path=/dev/sdc,Type=blockio
  Alias LUN1

But when I tried adding disks, I notice everything gets messed up (the initiators no longer recognizes some disks in their volume management, etc).

After a few digging, I found out the device IDs (/dev/sdX) change across reboots and when adding disks and modifying arrangements. By the way, Virtualbox offers SCSI, SATA, and IDE controllers. So in Linux, /dev/disk/by-uuid/ is the preferred way to reference disks but when I checked this directory, not all disks are represented in there. This is because only partitions with format have UUIDs. But no, I don't want my ISCSI target machine to format any disk, they should be presented to the initiators as block devices.

Basically my goal is to provision and present block devices to initiators, that would persist across changes and reboots. The initators will take care of managing the block devices (with any volume mgmt such as LVM or even ZFS and VxFS) and place filesystems on logical volumes. Any suggestions please? Any other approach?

Thanks!

smallpond · 03-18-2017, 10:01 AM

/dev/disk/by-id uses serial number info from the drive (virtual or physical) and is supposed to be unique and persistent across reboots.

ilesterg · 03-19-2017, 04:42 AM

Hi smallpond,

I did as you suggested, but it turns out in the end, the same scenario happened. After creating logical volumes (LVM) on the ISCSI initator, and rebooting both machines, the target will no longer be able to correctly present the disks.

Target:

Code:

[root@centrepo ~]# rpm -qa | grep scsi-target
scsi-target-utils-1.0.55-4.el7.x86_64
[root@centrepo ~]# rpm -qi scsi-target-utils
Name        : scsi-target-utils
Version     : 1.0.55
Release     : 4.el7
Architecture: x86_64
Install Date: Sat 04 Mar 2017 04:02:31 PM PHT
Group       : System Environment/Daemons
Size        : 541448
License     : GPLv2
Signature   : RSA/SHA256, Wed 13 Jan 2016 08:22:17 AM PHT, Key ID 6a2faea2352c64e5
Source RPM  : scsi-target-utils-1.0.55-4.el7.src.rpm
Build Date  : Wed 13 Jan 2016 04:08:27 AM PHT
Build Host  : buildvm-05-nfs.phx2.fedoraproject.org
Relocations : (not relocatable)
Packager    : Fedora Project
Vendor      : Fedora Project
URL         : http://stgt.sourceforge.net/
Summary     : The SCSI target daemon and utility programs
Description :
The SCSI target package contains the daemon and tools to setup a SCSI targets.
Currently, software iSCSI targets are supported.
[root@centrepo ~]# systemctl status -l tgtd
● tgtd.service - tgtd iSCSI target daemon
   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2017-03-19 16:53:27 PHT; 21min ago
  Process: 2534 ExecStartPost=/usr/sbin/tgtadm --op update --mode sys --name State -v ready (code=exited, status=0/SUCCESS)
  Process: 2492 ExecStartPost=/usr/sbin/tgt-admin -e -c $TGTD_CONFIG (code=exited, status=0/SUCCESS)
  Process: 2490 ExecStartPost=/usr/sbin/tgtadm --op update --mode sys --name State -v offline (code=exited, status=0/SUCCESS)
  Process: 1374 ExecStartPost=/bin/sleep 5 (code=exited, status=0/SUCCESS)
 Main PID: 1373 (tgtd)
   CGroup: /system.slice/tgtd.service
           └─1373 /usr/sbin/tgtd -f

Mar 19 16:53:20 centrepo.serverglcl.net systemd[1]: Starting tgtd iSCSI target daemon...
Mar 19 16:53:21 centrepo.serverglcl.net tgtd[1373]: tgtd: iser_ib_init(3436) Failed to initialize RDMA; load kernel modules?
Mar 19 16:53:21 centrepo.serverglcl.net tgtd[1373]: tgtd: work_timer_start(146) use timer_fd based scheduler
Mar 19 16:53:21 centrepo.serverglcl.net tgtd[1373]: tgtd: bs_init_signalfd(267) could not open backing-store module directory /usr/lib64/tgt/backing-store
Mar 19 16:53:21 centrepo.serverglcl.net tgtd[1373]: tgtd: bs_init(386) use signalfd notification
Mar 19 16:53:27 centrepo.serverglcl.net tgtd[1373]: tgtd: device_mgmt(246) sz:59 params:path=/dev/disk/by-id/ata-VBOX_HARDDISK_VB462ef870-7d53a83e
Mar 19 16:53:27 centrepo.serverglcl.net tgtd[1373]: tgtd: bs_thread_open(408) 16
Mar 19 16:53:27 centrepo.serverglcl.net systemd[1]: Started tgtd iSCSI target daemon.
[root@centrepo ~]#

I suspect this has something to do with the fact that LVM headers are already writted to the disks. When you run LVM commands on the target, you can see the LVM records which I created on the initiator:

Code:

[root@centrepo ~]# lvs
  LV         VG         Attr       LSize   Pool Origin Data%  Meta%  Move Log               Cpy%Sync Convert
  lv_root    vg00       -wi-ao----   3.79g                                                                  
  lvdboxacct vgdropbox  mwi-a-m--- 512.00m                                [lvdboxacct_mlog] 100.00          
  lvvideos   vghomework -wi-a-----  45.00g                                                                  
  lv68       vgrepos    -wi-ao----  20.00g                                                                  
  lv73       vgrepos    -wi-ao----  20.00g                                                                  
[root@centrepo ~]#

Also, you will notice in the below output that targets 1-5 only have LUN0 and does not display any backing store path. All of these already have LVM headers on them (i.e. I already used them on the initiator as PVs). But the last one, target 6, has LUN1 and a target path. This is the only one I still haven't used on the initiator.

Code:

[root@centrepo ~]# tgtadm --mode target --op show
Target 1: iqn.2017-03.net.serverglcl:datadisk1
    System information:
        Driver: iscsi
        State: ready
    I_T nexus information:
        I_T nexus: 3
            Initiator: iqn.2017-03.net.serverglcl.centserver:5daf2044cb2c alias: centserver.serverglcl.net
            Connection: 0
                IP Address: 10.0.2.6
        I_T nexus: 4
            Initiator: iqn.2017-03.net.serverglcl.centserver:5daf2044cb2c alias: centserver.serverglcl.net
            Connection: 0
                IP Address: 10.0.3.6
    LUN information:
        LUN: 0
            Type: controller
            SCSI ID: IET     00010000
            SCSI SN: beaf10
            Size: 0 MB, Block size: 1
            Online: Yes
            Removable media: No
            Prevent removal: No
            Readonly: No
            SWP: No
            Thin-provisioning: No
            Backing store type: null
            Backing store path: None
            Backing store flags: 
    Account information:
    ACL information:
        ALL
Target 2: iqn.2017-03.net.serverglcl:datadisk2
    System information:
        Driver: iscsi
        State: ready
    I_T nexus information:
        I_T nexus: 5
            Initiator: iqn.2017-03.net.serverglcl.centserver:5daf2044cb2c alias: centserver.serverglcl.net
            Connection: 0
                IP Address: 10.0.2.6
        I_T nexus: 6
            Initiator: iqn.2017-03.net.serverglcl.centserver:5daf2044cb2c alias: centserver.serverglcl.net
            Connection: 0
                IP Address: 10.0.3.6
    LUN information:
        LUN: 0
            Type: controller
            SCSI ID: IET     00020000
            SCSI SN: beaf20
            Size: 0 MB, Block size: 1
            Online: Yes
            Removable media: No
            Prevent removal: No
            Readonly: No
            SWP: No
            Thin-provisioning: No
            Backing store type: null
            Backing store path: None
            Backing store flags: 
    Account information:
    ACL information:
        ALL
Target 3: iqn.2017-03.net.serverglcl:datadisk3
    System information:
        Driver: iscsi
        State: ready
    I_T nexus information:
    LUN information:
        LUN: 0
            Type: controller
            SCSI ID: IET     00030000
            SCSI SN: beaf30
            Size: 0 MB, Block size: 1
            Online: Yes
            Removable media: No
            Prevent removal: No
            Readonly: No
            SWP: No
            Thin-provisioning: No
            Backing store type: null
            Backing store path: None
            Backing store flags: 
    Account information:
    ACL information:
        ALL
Target 4: iqn.2017-03.net.serverglcl:dropbox1
    System information:
        Driver: iscsi
        State: ready
    I_T nexus information:
    LUN information:
        LUN: 0
            Type: controller
            SCSI ID: IET     00040000
            SCSI SN: beaf40
            Size: 0 MB, Block size: 1
            Online: Yes
            Removable media: No
            Prevent removal: No
            Readonly: No
            SWP: No
            Thin-provisioning: No
            Backing store type: null
            Backing store path: None
            Backing store flags: 
    Account information:
    ACL information:
        ALL
Target 5: iqn.2017-03.net.serverglcl:dropbox2
    System information:
        Driver: iscsi
        State: ready
    I_T nexus information:
    LUN information:
        LUN: 0
            Type: controller
            SCSI ID: IET     00050000
            SCSI SN: beaf50
            Size: 0 MB, Block size: 1
            Online: Yes
            Removable media: No
            Prevent removal: No
            Readonly: No
            SWP: No
            Thin-provisioning: No
            Backing store type: null
            Backing store path: None
            Backing store flags: 
    Account information:
    ACL information:
        ALL
Target 6: iqn.2017-03.net.serverglcl:sbd1
    System information:
        Driver: iscsi
        State: ready
    I_T nexus information:
    LUN information:
        LUN: 0
            Type: controller
            SCSI ID: IET     00060000
            SCSI SN: beaf60
            Size: 0 MB, Block size: 1
            Online: Yes
            Removable media: No
            Prevent removal: No
            Readonly: No
            SWP: No
            Thin-provisioning: No
            Backing store type: null
            Backing store path: None
            Backing store flags: 
        LUN: 1
            Type: disk
            SCSI ID: IET     00060001
            SCSI SN: beaf61
            Size: 1074 MB, Block size: 512
            Online: Yes
            Removable media: No
            Prevent removal: No
            Readonly: No
            SWP: No
            Thin-provisioning: No
            Backing store type: rdwr
            Backing store path: /dev/disk/by-id/ata-VBOX_HARDDISK_VB462ef870-7d53a83e
            Backing store flags: 
    Account information:
    ACL information:
        ALL
[root@centrepo ~]#

True enough, if I log in to this target 6 on the initiator, I will be able to:

Code:

[root@centserver ~]# iscsiadm --mode node --targetname iqn.2017-03.net.serverglcl:sbd1 --portal 10.0.2.4 --login
Logging in to [iface: default, target: iqn.2017-03.net.serverglcl:sbd1, portal: 10.0.2.4,3260] (multiple)
Login to [iface: default, target: iqn.2017-03.net.serverglcl:sbd1, portal: 10.0.2.4,3260] successful.
[root@centserver ~]# iscsiadm --mode node --targetname iqn.2017-03.net.serverglcl:sbd1 --portal 10.0.3.4 --login
Logging in to [iface: default, target: iqn.2017-03.net.serverglcl:sbd1, portal: 10.0.3.4,3260] (multiple)
Login to [iface: default, target: iqn.2017-03.net.serverglcl:sbd1, portal: 10.0.3.4,3260] successful.
[root@centserver ~]# lsscsi
[0:0:0:0]    cd/dvd  VBOX     CD-ROM           1.0   /dev/sr0 
[2:0:0:0]    disk    ATA      VBOX HARDDISK    1.0   /dev/sda 
[17:0:0:0]   storage IET      Controller       0001  -        
[18:0:0:0]   storage IET      Controller       0001  -        
[19:0:0:0]   storage IET      Controller       0001  -        
[20:0:0:0]   storage IET      Controller       0001  -        
[21:0:0:0]   storage IET      Controller       0001  -        
[21:0:0:1]   disk    IET      VIRTUAL-DISK     0001  /dev/sdb 
[22:0:0:0]   storage IET      Controller       0001  -        
[22:0:0:1]   disk    IET      VIRTUAL-DISK     0001  /dev/sdc 
[root@centserver ~]# 
[root@centserver ~]# multipath -ll
mpatht (360000000000000000e00000000060001) dm-2 IET     ,VIRTUAL-DISK    
size=1.0G features='0' hwhandler='0' wp=rw
|-+- policy='service-time 0' prio=1 status=active
| `- 22:0:0:1 sdc 8:32 active ready running
`-+- policy='service-time 0' prio=1 status=enabled
  `- 21:0:0:1 sdb 8:16 active ready running
[root@centserver ~]#

Also, I have no idea why target 6 has LUN1.

Code:

[root@centrepo ~]# fdisk -l /dev/disk/by-id/ata-VBOX_HARDDISK_VB462ef870-7d53a83e

Disk /dev/disk/by-id/ata-VBOX_HARDDISK_VB462ef870-7d53a83e: 1073 MB, 1073741824 bytes, 2097152 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

[root@centrepo ~]#

Thanks.

ilesterg · 03-19-2017, 04:44 AM

Ok so I noticed something. LUN0 is the controller, and LUN1 is the disk. This would make sense on the initiator:

Quote:

[root@centserver ~]# lsscsi
[0:0:0:0] cd/dvd VBOX CD-ROM 1.0 /dev/sr0
[2:0:0:0] disk ATA VBOX HARDDISK 1.0 /dev/sda
[17:0:0:0] storage IET Controller 0001 -
[18:0:0:0] storage IET Controller 0001 -
[19:0:0:0] storage IET Controller 0001 -
[20:0:0:0] storage IET Controller 0001 -
[21:0:0:0] storage IET Controller 0001 -
[21:0:0:1] disk IET VIRTUAL-DISK 0001 /dev/sdb
[22:0:0:0] storage IET Controller 0001 -
[22:0:0:1] disk IET VIRTUAL-DISK 0001 /dev/sdc
[root@centserver ~]#

-> 17:0:0:0 to 20:0:0:0 are all controllers only, no disks.

How can I check why the target fails to present the disks at all?

Thanks.