LinuxQuestions.org
Welcome to the most active Linux Forum on the web.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 03-18-2017, 04:32 AM   #1
ilesterg
Member
 
Registered: Jul 2012
Location: München
Distribution: Debian, CentOS/RHEL
Posts: 587

Rep: Reputation: 72
iscsi target | Persistent disk targets


Hi!

This question goes for Debian based and RHEL based distros, but I believe it will apply to other distros as well.

I'm working on a Virtulbox environment, and I have been trying to set up iscsi targets to mimic a "SAN" and have been successful so far. On a variety of targets, I was able to format the disks using a variety of combinations: LVM mirrored, striped, different types of filesystems etc.

This is how devices are represented in my target's configuration:
Code:
Target iqn.2017-02.net.lcldomain:lun68_1
  Lun 0 Path=/dev/sdb,Type=blockio
  Alias LUN0
Target iqn.2017-02.net.lcldomain:lun73_1
  Lun 1 Path=/dev/sdc,Type=blockio
  Alias LUN1
But when I tried adding disks, I notice everything gets messed up (the initiators no longer recognizes some disks in their volume management, etc).

After a few digging, I found out the device IDs (/dev/sdX) change across reboots and when adding disks and modifying arrangements. By the way, Virtualbox offers SCSI, SATA, and IDE controllers. So in Linux, /dev/disk/by-uuid/ is the preferred way to reference disks but when I checked this directory, not all disks are represented in there. This is because only partitions with format have UUIDs. But no, I don't want my ISCSI target machine to format any disk, they should be presented to the initiators as block devices.

Basically my goal is to provision and present block devices to initiators, that would persist across changes and reboots. The initators will take care of managing the block devices (with any volume mgmt such as LVM or even ZFS and VxFS) and place filesystems on logical volumes. Any suggestions please? Any other approach?

Thanks!

Last edited by ilesterg; 03-18-2017 at 05:08 AM.
 
Old 03-18-2017, 10:01 AM   #2
smallpond
Senior Member
 
Registered: Feb 2011
Location: Massachusetts, USA
Distribution: Fedora 33
Posts: 3,598

Rep: Reputation: 1027Reputation: 1027Reputation: 1027Reputation: 1027Reputation: 1027Reputation: 1027Reputation: 1027Reputation: 1027
/dev/disk/by-id uses serial number info from the drive (virtual or physical) and is supposed to be unique and persistent across reboots.
 
Old 03-19-2017, 04:42 AM   #3
ilesterg
Member
 
Registered: Jul 2012
Location: München
Distribution: Debian, CentOS/RHEL
Posts: 587

Original Poster
Rep: Reputation: 72
Hi smallpond,

I did as you suggested, but it turns out in the end, the same scenario happened. After creating logical volumes (LVM) on the ISCSI initator, and rebooting both machines, the target will no longer be able to correctly present the disks.

Target:
Code:
[root@centrepo ~]# rpm -qa | grep scsi-target
scsi-target-utils-1.0.55-4.el7.x86_64
[root@centrepo ~]# rpm -qi scsi-target-utils
Name        : scsi-target-utils
Version     : 1.0.55
Release     : 4.el7
Architecture: x86_64
Install Date: Sat 04 Mar 2017 04:02:31 PM PHT
Group       : System Environment/Daemons
Size        : 541448
License     : GPLv2
Signature   : RSA/SHA256, Wed 13 Jan 2016 08:22:17 AM PHT, Key ID 6a2faea2352c64e5
Source RPM  : scsi-target-utils-1.0.55-4.el7.src.rpm
Build Date  : Wed 13 Jan 2016 04:08:27 AM PHT
Build Host  : buildvm-05-nfs.phx2.fedoraproject.org
Relocations : (not relocatable)
Packager    : Fedora Project
Vendor      : Fedora Project
URL         : http://stgt.sourceforge.net/
Summary     : The SCSI target daemon and utility programs
Description :
The SCSI target package contains the daemon and tools to setup a SCSI targets.
Currently, software iSCSI targets are supported.
[root@centrepo ~]# systemctl status -l tgtd
● tgtd.service - tgtd iSCSI target daemon
   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2017-03-19 16:53:27 PHT; 21min ago
  Process: 2534 ExecStartPost=/usr/sbin/tgtadm --op update --mode sys --name State -v ready (code=exited, status=0/SUCCESS)
  Process: 2492 ExecStartPost=/usr/sbin/tgt-admin -e -c $TGTD_CONFIG (code=exited, status=0/SUCCESS)
  Process: 2490 ExecStartPost=/usr/sbin/tgtadm --op update --mode sys --name State -v offline (code=exited, status=0/SUCCESS)
  Process: 1374 ExecStartPost=/bin/sleep 5 (code=exited, status=0/SUCCESS)
 Main PID: 1373 (tgtd)
   CGroup: /system.slice/tgtd.service
           └─1373 /usr/sbin/tgtd -f

Mar 19 16:53:20 centrepo.serverglcl.net systemd[1]: Starting tgtd iSCSI target daemon...
Mar 19 16:53:21 centrepo.serverglcl.net tgtd[1373]: tgtd: iser_ib_init(3436) Failed to initialize RDMA; load kernel modules?
Mar 19 16:53:21 centrepo.serverglcl.net tgtd[1373]: tgtd: work_timer_start(146) use timer_fd based scheduler
Mar 19 16:53:21 centrepo.serverglcl.net tgtd[1373]: tgtd: bs_init_signalfd(267) could not open backing-store module directory /usr/lib64/tgt/backing-store
Mar 19 16:53:21 centrepo.serverglcl.net tgtd[1373]: tgtd: bs_init(386) use signalfd notification
Mar 19 16:53:27 centrepo.serverglcl.net tgtd[1373]: tgtd: device_mgmt(246) sz:59 params:path=/dev/disk/by-id/ata-VBOX_HARDDISK_VB462ef870-7d53a83e
Mar 19 16:53:27 centrepo.serverglcl.net tgtd[1373]: tgtd: bs_thread_open(408) 16
Mar 19 16:53:27 centrepo.serverglcl.net systemd[1]: Started tgtd iSCSI target daemon.
[root@centrepo ~]#
I suspect this has something to do with the fact that LVM headers are already writted to the disks. When you run LVM commands on the target, you can see the LVM records which I created on the initiator:
Code:
[root@centrepo ~]# lvs
  LV         VG         Attr       LSize   Pool Origin Data%  Meta%  Move Log               Cpy%Sync Convert
  lv_root    vg00       -wi-ao----   3.79g                                                                  
  lvdboxacct vgdropbox  mwi-a-m--- 512.00m                                [lvdboxacct_mlog] 100.00          
  lvvideos   vghomework -wi-a-----  45.00g                                                                  
  lv68       vgrepos    -wi-ao----  20.00g                                                                  
  lv73       vgrepos    -wi-ao----  20.00g                                                                  
[root@centrepo ~]#
Also, you will notice in the below output that targets 1-5 only have LUN0 and does not display any backing store path. All of these already have LVM headers on them (i.e. I already used them on the initiator as PVs). But the last one, target 6, has LUN1 and a target path. This is the only one I still haven't used on the initiator.
Code:
[root@centrepo ~]# tgtadm --mode target --op show
Target 1: iqn.2017-03.net.serverglcl:datadisk1
    System information:
        Driver: iscsi
        State: ready
    I_T nexus information:
        I_T nexus: 3
            Initiator: iqn.2017-03.net.serverglcl.centserver:5daf2044cb2c alias: centserver.serverglcl.net
            Connection: 0
                IP Address: 10.0.2.6
        I_T nexus: 4
            Initiator: iqn.2017-03.net.serverglcl.centserver:5daf2044cb2c alias: centserver.serverglcl.net
            Connection: 0
                IP Address: 10.0.3.6
    LUN information:
        LUN: 0
            Type: controller
            SCSI ID: IET     00010000
            SCSI SN: beaf10
            Size: 0 MB, Block size: 1
            Online: Yes
            Removable media: No
            Prevent removal: No
            Readonly: No
            SWP: No
            Thin-provisioning: No
            Backing store type: null
            Backing store path: None
            Backing store flags: 
    Account information:
    ACL information:
        ALL
Target 2: iqn.2017-03.net.serverglcl:datadisk2
    System information:
        Driver: iscsi
        State: ready
    I_T nexus information:
        I_T nexus: 5
            Initiator: iqn.2017-03.net.serverglcl.centserver:5daf2044cb2c alias: centserver.serverglcl.net
            Connection: 0
                IP Address: 10.0.2.6
        I_T nexus: 6
            Initiator: iqn.2017-03.net.serverglcl.centserver:5daf2044cb2c alias: centserver.serverglcl.net
            Connection: 0
                IP Address: 10.0.3.6
    LUN information:
        LUN: 0
            Type: controller
            SCSI ID: IET     00020000
            SCSI SN: beaf20
            Size: 0 MB, Block size: 1
            Online: Yes
            Removable media: No
            Prevent removal: No
            Readonly: No
            SWP: No
            Thin-provisioning: No
            Backing store type: null
            Backing store path: None
            Backing store flags: 
    Account information:
    ACL information:
        ALL
Target 3: iqn.2017-03.net.serverglcl:datadisk3
    System information:
        Driver: iscsi
        State: ready
    I_T nexus information:
    LUN information:
        LUN: 0
            Type: controller
            SCSI ID: IET     00030000
            SCSI SN: beaf30
            Size: 0 MB, Block size: 1
            Online: Yes
            Removable media: No
            Prevent removal: No
            Readonly: No
            SWP: No
            Thin-provisioning: No
            Backing store type: null
            Backing store path: None
            Backing store flags: 
    Account information:
    ACL information:
        ALL
Target 4: iqn.2017-03.net.serverglcl:dropbox1
    System information:
        Driver: iscsi
        State: ready
    I_T nexus information:
    LUN information:
        LUN: 0
            Type: controller
            SCSI ID: IET     00040000
            SCSI SN: beaf40
            Size: 0 MB, Block size: 1
            Online: Yes
            Removable media: No
            Prevent removal: No
            Readonly: No
            SWP: No
            Thin-provisioning: No
            Backing store type: null
            Backing store path: None
            Backing store flags: 
    Account information:
    ACL information:
        ALL
Target 5: iqn.2017-03.net.serverglcl:dropbox2
    System information:
        Driver: iscsi
        State: ready
    I_T nexus information:
    LUN information:
        LUN: 0
            Type: controller
            SCSI ID: IET     00050000
            SCSI SN: beaf50
            Size: 0 MB, Block size: 1
            Online: Yes
            Removable media: No
            Prevent removal: No
            Readonly: No
            SWP: No
            Thin-provisioning: No
            Backing store type: null
            Backing store path: None
            Backing store flags: 
    Account information:
    ACL information:
        ALL
Target 6: iqn.2017-03.net.serverglcl:sbd1
    System information:
        Driver: iscsi
        State: ready
    I_T nexus information:
    LUN information:
        LUN: 0
            Type: controller
            SCSI ID: IET     00060000
            SCSI SN: beaf60
            Size: 0 MB, Block size: 1
            Online: Yes
            Removable media: No
            Prevent removal: No
            Readonly: No
            SWP: No
            Thin-provisioning: No
            Backing store type: null
            Backing store path: None
            Backing store flags: 
        LUN: 1
            Type: disk
            SCSI ID: IET     00060001
            SCSI SN: beaf61
            Size: 1074 MB, Block size: 512
            Online: Yes
            Removable media: No
            Prevent removal: No
            Readonly: No
            SWP: No
            Thin-provisioning: No
            Backing store type: rdwr
            Backing store path: /dev/disk/by-id/ata-VBOX_HARDDISK_VB462ef870-7d53a83e
            Backing store flags: 
    Account information:
    ACL information:
        ALL
[root@centrepo ~]#


True enough, if I log in to this target 6 on the initiator, I will be able to:
Code:
[root@centserver ~]# iscsiadm --mode node --targetname iqn.2017-03.net.serverglcl:sbd1 --portal 10.0.2.4 --login
Logging in to [iface: default, target: iqn.2017-03.net.serverglcl:sbd1, portal: 10.0.2.4,3260] (multiple)
Login to [iface: default, target: iqn.2017-03.net.serverglcl:sbd1, portal: 10.0.2.4,3260] successful.
[root@centserver ~]# iscsiadm --mode node --targetname iqn.2017-03.net.serverglcl:sbd1 --portal 10.0.3.4 --login
Logging in to [iface: default, target: iqn.2017-03.net.serverglcl:sbd1, portal: 10.0.3.4,3260] (multiple)
Login to [iface: default, target: iqn.2017-03.net.serverglcl:sbd1, portal: 10.0.3.4,3260] successful.
[root@centserver ~]# lsscsi
[0:0:0:0]    cd/dvd  VBOX     CD-ROM           1.0   /dev/sr0 
[2:0:0:0]    disk    ATA      VBOX HARDDISK    1.0   /dev/sda 
[17:0:0:0]   storage IET      Controller       0001  -        
[18:0:0:0]   storage IET      Controller       0001  -        
[19:0:0:0]   storage IET      Controller       0001  -        
[20:0:0:0]   storage IET      Controller       0001  -        
[21:0:0:0]   storage IET      Controller       0001  -        
[21:0:0:1]   disk    IET      VIRTUAL-DISK     0001  /dev/sdb 
[22:0:0:0]   storage IET      Controller       0001  -        
[22:0:0:1]   disk    IET      VIRTUAL-DISK     0001  /dev/sdc 
[root@centserver ~]# 
[root@centserver ~]# multipath -ll
mpatht (360000000000000000e00000000060001) dm-2 IET     ,VIRTUAL-DISK    
size=1.0G features='0' hwhandler='0' wp=rw
|-+- policy='service-time 0' prio=1 status=active
| `- 22:0:0:1 sdc 8:32 active ready running
`-+- policy='service-time 0' prio=1 status=enabled
  `- 21:0:0:1 sdb 8:16 active ready running
[root@centserver ~]#
Also, I have no idea why target 6 has LUN1.
Code:
[root@centrepo ~]# fdisk -l /dev/disk/by-id/ata-VBOX_HARDDISK_VB462ef870-7d53a83e

Disk /dev/disk/by-id/ata-VBOX_HARDDISK_VB462ef870-7d53a83e: 1073 MB, 1073741824 bytes, 2097152 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

[root@centrepo ~]#
Thanks.
 
Old 03-19-2017, 04:44 AM   #4
ilesterg
Member
 
Registered: Jul 2012
Location: München
Distribution: Debian, CentOS/RHEL
Posts: 587

Original Poster
Rep: Reputation: 72
Ok so I noticed something. LUN0 is the controller, and LUN1 is the disk. This would make sense on the initiator:
Quote:
[root@centserver ~]# lsscsi
[0:0:0:0] cd/dvd VBOX CD-ROM 1.0 /dev/sr0
[2:0:0:0] disk ATA VBOX HARDDISK 1.0 /dev/sda
[17:0:0:0] storage IET Controller 0001 -
[18:0:0:0] storage IET Controller 0001 -
[19:0:0:0] storage IET Controller 0001 -
[20:0:0:0] storage IET Controller 0001 -
[21:0:0:0] storage IET Controller 0001 -
[21:0:0:1] disk IET VIRTUAL-DISK 0001 /dev/sdb
[22:0:0:0] storage IET Controller 0001 -
[22:0:0:1] disk IET VIRTUAL-DISK 0001 /dev/sdc
[root@centserver ~]#
-> 17:0:0:0 to 20:0:0:0 are all controllers only, no disks.

How can I check why the target fails to present the disks at all?

Thanks.
 
  


Reply

Tags
iscsi, lvm, san, storage


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] pacemaker - iscsi: how to set up iscsi targets/logical units? eantoranz Linux - Server 9 01-02-2013 08:38 AM
Persistent mount of a Logical Volume on a iscsi disk hup Red Hat 2 12-19-2012 03:46 PM
Loading iscsi targets on reboot Neruocomp *BSD 0 06-28-2010 10:03 AM
LVM doesent updates pvsize if pv is iscsi disk and size is increased on target side devkpict Linux - Software 3 07-06-2009 11:01 AM
changes on other iscsi nodes attached to an ext3 iscsi target not being seen sldahlin Linux - Server 1 06-07-2008 02:02 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie

All times are GMT -5. The time now is 04:42 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration