Hello all!

I am having trouble PXE booting a CentOS client in VMware which doesn't support iSCSI booting. I have set up a Debian (Wheezy) SAN to boot a CentOS6 client in VMWare Workstation 9 as a project for work. VMWare Workstation 9 doesn't support iSCSI boot, but it does support PXE, so I have resorted to PXE chainloading.

The Debian server has bind9, isc-dhcp-server and tftpd-hpa configured to serve a DHCP address the the client and TFTP boot an iPXE file (undionly.kpxe) that will give the ability to boot iSCSI. Everything works great as CentOS boots, but errors on connecting to the SAN.

I took extensive notes for those of you who are either extremely thorough or gluttons for punishment! My notes are below, the problem seems to be in the syntax used in /etc/dhcp/dhcpd.conf "iscsi:initiator1.example.com::::iqn.2013-06.com.example:storage.lun0"

Anybody here have an idea of what my mistake may be? I have added a .png screenshot of the error as CentOS loads attached if you'd like to see the error.

Thanks in advance for your time! And here comes the pain:

// Another day, another attempt at configuring this!
// Reverted back to a snapshot of a freshly installed Debian "san0"
// going to set up Bind9 first, make sure there are no loopholes in my topology...

apt-get install bind9

vim /etc/resolv.conf
search example.com
nameserver 192.168.99.6

chattr +i /etc/resolv.conf
// A new command I learned to make this file unwritable/delete-able by anyone!
// chattr -i {file} to remove this attribute, write to file and then chattr +i {file} again to lock it

vim /etc/network/interfaces
# San to Client interface
auto eth1
iface eth1 inet static
address 192.168.99.6
netmask 255.255.255.0
network 192.168.99.0
nameserver 192.168.99.6

cp /etc/bind/named.conf.local named.conf.local_ORIGINAL
cp /etc/bind/db.local example.com
cp /etc/bind/db.127 example.com.loopback

vim /etc/bind/named.conf.local
# San to Client zone
zone "example.com" IN {
Type master;
file "/etc/bind/example.com";
allow-update {none;};
};

# Reverse-lookup zone
zone "99.168.192.in-addr.arpa" IN {
Type master;
file "/etc/bind/example.com.loopback";
allow-update {none;};
};

vim /etc/bind/example.com
@ IN SOA example.com. root.example.com. (
{arbitrary stuff written here...}

;
@ IN NS example.com.
@ IN A 192.168.99.6
@ IN MX 0 mail

san0 IN A 192.168.99.6
san1 IN A 192.168.99.5
initiator1 IN A 192.168.99.10
initiator2 IN A 192.168.99.11

vim /etc/bind/example.com.loopback
@ IN SOA san0.example.com. root.example.com. (
{arbitrary stuff written here...}

;
99.168.192.in-addr.arpa. IN NS san0.example.com.

6 IN PTR san0.example.com
5 IN PTR san1.example.com
10 IN PTR initiator1.example.com.
11 IN PTR initiator2.example.com.

nslookup example.com
Server: 192.168.99.6
Address: 192.168.99.6#53
Name:example.com
Address: 192.168.99.6

nslookup initiator1
Server: 192.168.99.6
Address 192.168.99.6#53
Name: initiator1.example.com
Address: 192.168.99.10

nslookup 192.168.99.11
Server: 192.168.99.6
Address 192.168.99.6#53
11.99.168.192.in-addr.arpa name = initiator2.example.com

// Sweet, looks like our DNS is working like it should!
// A minor success, but a success nonetheless!
// Time to take a break and work on DHCP after the wife goes to bed...


// Ok, it's only 1030pm... sart with setting up DHCP
// Should I use isc-dhcp-server or dnsmasq?
// It looks like dnsmasq can double as dhcp and tftp, but will its dns affect bind9?
// Let's just use isc-dhcp-server since we know that one...
// But let's reinstall CentOS to our iSCSI Target image & find the undionly.kpxe we need
// Uh, let's make a snapshot since even awesome notes can't save me sometimes! (san0_1)

w3m http://etherboot.org/wiki/pxechaining

// Ok, found a preconfigured file from rom-o-matic (fingers crossed...)
// It's in root (/) and is named "gpxe-1.0.1-undionly.kpxe"

mkdir /storage
dd if=/dev/zero of=/storage/lun0.img bs=1024 count=5000000
dd if=/dev/zero of=/storage/lun1.img bs=1024 count=5000000
cd /storage
ls -alh
4.8G lun0.img
4.8G lun1.img

// And we're gonna' need our iscsi target running to install...

apt-get install iscsitarget iscsitarget-dkms
vim /etc/default/iscsitarget
ISCSITARGET_ENABLE=true

vim /etc/iet/ietd.conf
#
#iSCSI connection to Client (initiator1 @ 192.168.99.10)
Target iqn.2013-06.com.example:storage.lun0
Lun 0 Path=/storage/lun0.img
Alias LUN0

reboot
// Just for good measure...

service isc-dhcp-server status
Running
service iscsitarget status
Running

// Now over to the CentOS install...

eth0 is configured DHCP to Comcast to get the netinstall
http://mirror.centos.org/centos/6/os/x86_64
eth1 is configured static to SAN0 (192.168.99.6)
Advanced Storage Options:
Add iSCSI target
Bind targets to network interfaces
Target IP: 192.168.99.6
iSCSI Initiator Name: initiator1
Vendor: iet
Interconnect: SCSI
Device: mpatha
Hostname:initiator1

// Ok, now that our extra stuff is ready(?), we can install CentOS again & work on DHCP


vim /etc/dhcp/dhcpd/conf
ddns-update-style none;
option domain-name "example.com";
option domain-name-servers san0.example.com;
default-lease-time 6000;
max-lease-time 7200;
authorative;
subnet 192.168.99.0 netmask 255.255.255.0 {
range 192.168.99.15 192.168.99.20;
}
//Put this in because DHCP was complaining about having no declaration for eth1...

host initiator1 {
hardware ethernet 00:0C:29:A8:93F;
fixed-address 192.168.99.10;
}

if exists user-class and option user-class = "iPXE" {
filename "";
option root-path "iscsi:initiator1.example.com::::iqn.2013-06.com.example:storage.lun0";
} else {
next-server 192.168.99.6;
filename "undionly.kpxe";
}

service isc-dhcp-server restart

// After installing CentOS, it looks like it is getting an address and looking for a TFTP server
// Unfortunately it's now almost 3am... time to finish tomorrow.
'
'
// Ok, time to look for 2am spelling errors, now that it's Monday morning!
// Take another snapshot (san0_2)

apt-get install tftpd-hpa

wget http://boot.ipxe.org/undionly.kpxe
// The older gPXE I got from etherboot isn't supported, get an iPXE instead

vim /etc/default/tftpd-hpa
TFTP_USERNAME="tftp"
TFTP_DIRECTORY="/srv/tftp"
TFTP_ADDRESS="0.0.0.0:69"
TFTP_OPTIONS="--secure"

cp undionly.kpxe /srv/tftp/undionly.kpxe

service tftpd-hpa restart
service tftpd-hpa status
// Running
service isc-dhcp-server restart
service isc-dhcp-server status
// Running

// Time to boot the client again...
// Limited success! Boots to PXE, finds a TFTP server and boots iPXE but then errors out...
// "Could not open SAN device: Connection timed out"
// It looks like my syntax:
Root path: iscsi:initiator1.example.com::::iqn.2013-06.com.example:storage.lun0
// May be incorrect.... it's Monday, lets see what Abe thinks of this error...

Attached Thumbnails

 

In your DHCP config, have you tried using an IP address instead of DNS name in the root path?

option root-path "iscsi:initiator1.example.com::::iqn.2013-06.com.example:storage.lun0";

You seem to have two network cards in the iPXE client machine, one connecting to your ISP and another connecting to an internal network. Are you sure iPXE is actually getting the DHCP response on the NIC with the iSCSI target and not the other one? My guess is that you might be getting the wrong one, and you're trying to send out an iSCSI request on the wrong network, for some reason.

You could try my embedded script located at https://gist.github.com/robinsmidsrod/3871687 which allows you to explicitly choose which NIC to boot from. Follow the instructions on the top for how to compile it, or use http://rom-o-matic.eu/ which can create an iPXE binary for you just by pasting in the script.

Best regards,
Robin Smidsrød

Hey buddy, thanks for the response! Much appreciated.

"In your DHCP config, have you tried using an IP address instead of DNS name in the root path?"

Yep, sure have, didn't make a difference. Connection still timed out while trying to get a connection to the iSCSI target, which was disappointing since CentOS install found the iSCSI target and installed to it with no problems. DHCP on two different ports shouldn't be the issue since DHCP is being handed out on both ends by IP address reservation (MAC address) and it shows on boot that it is getting the correct address on the correct interface complete with domain, next-server address and the root-path...

I even got in contact with users on VMware & iPXE forums who helped me tweak the config better but still to the same effect. And they are all in agreement over that the config "should" work and is correct. I'll post the changed configs if you're interested in seeing them, I'd love to get an actual answer on the problem!

As it is, the only problem I can find is that: as it is an off-the-clock project and I'm a lowly intern I doing this all on a Windows laptop running VMWare Workstation 9 and that Workstation's BIOS is buggy and it just may not work... or so I've been told. But that doesn't sound like a real answer, does it? Just a maybe.

I will check out that embedded script tomorrow and much appreciated for the advice. As is, even the "project manager" has concluded he doesn't see why my configs are failing and my new task is install CentOS with KVM and get a few machines up and running in it. That shouldn't be too hard (**famous last words**)!

I was going to link to the iPXE forum post, but I can't seem to find it, even though I seem to recall it was one I responded to... Could you help me out?

You won't find it on there anymore... I accidentally double posted on there and, when I tried to delete one of the duplicate postings, I managed to delete the thread entirely. I even deleted the extremely long response I gave... Ugh! Fortunately I had the forum open on my iPhone too, so I pulled all the comments off and plan to reopen the post. I'll do that shortly and I will send the link if you'd like. Thanks for the response, always nice to hear from ya'!

"I seem to recall it was one I responded to... "

Does that mean this is "robinsmidsrod" from the iPXE forum group?

Indeed it is. Didn't you see my signature in my first response? I'm everywhere! Google Alerts to the rescue.

Awesome! You tried to help on the iPXE forum and I wrote this long response thanking you and showing all the different configs I tried... then I didn't like how long the thread had gotten with the double posting, tried to delete one of the posts and promptly deleted the whole thing! Argh!

Anyways, thanks so much for trying to help me on the iPXE forum! I sure do appreciate it, though it's awesome to be able to speak with you about it here as well! All-in-all it is my belief that it actually ended up being a VMWare problem and not an iPXE issue. I showed my configs to my boss, my co-workers, my teachers from school (even though I already graduated! Kudos to them for being so damn helpful!), the iPXE forum, this LQ forum and even the VMWare forum and couldn't get an answer... fact is, I used the same configs on a few throw-away computers at work and got it running just fine.

If it helps to know, I even saved your response as I got lucky and had it open on my iPhone's browser after I had deleted it from the forum. Here is your posting:
***************************************

To be honest, this looks like a proper configuration. You might want to put those three ipxe-specific DHCP options inside the if-userclass-ipxe conditional, they are not very useful outside the block.

My best bet is for you to enable syslog output (see http://ipxe.org/console for details) so you can understand where things go wrong.

Also, you say
Quote:
Connection fails with: Could not open SAN device: connection timed out.
. There should be a URL associated with that error. Did you actually open that in a browser and read what it said? Can you post that error URL you got as well? My guess is that you're unable to connect to your iSCSI target for some reason. I would try a packet capture to figure out exactly what's going on.

If you need more details you could try to compile iPXE with DEBUG=iscsi,scsi,int13. That should give you more details to share if you're unable to figure things out. This will create a lot of output, so you'll need the syslog output (or serial) to be able to capture it properly.
******************************************

In response, yes, I did check the error message url it gave:

Could not open SAN device: Connection timed out (http://ipxe.org/4c0a6035)

Which point to it being an error in

net/tcp.c (line 658)

But when I look for that line, and all the lines around it, it is just coding for an error message. I didn't manage to gain any info that would help me to troubleshoot the problem. The closest thing I got concerning what may be wrong was on the VMWare forum, where it was stated that it could be a problem with VMWare Workstation9's "buggy BIOS". That's not really an answer, though... just a hypothetical.

If you are interested in seeing my current configs I would be more than happy to post them here (or on iPXE forum). I'm at work at the moment, so I must finish this out, but thanks much for taking the time to help! I hope to have the opportunity to talk with you more about it. I'm excited about the possibilities of using iPXE in a production environment and you can be sure to be hearing more from me in the future! Have a nice one, buddy!

Here is a screenshot (after many configuration changes) of the error message I was getting:

Attached Thumbnails

 

That screenshot doesn't contain any DEBUG information. Try to get either serial or syslog logging working, then it should be easier to see what it says. And I'd urge you to continue the conversation on the iPXE forum, as more netboot-experienced users are there than here (I guess).