Hi. I'm jon.404, a Unix/Linux/Database/Openstack/Kubernetes Administrator, AWS/GCP/Azure Engineer, mathematics enthusiast, and amateur philosopher. This is where I rant about that which upsets me, laugh about that which amuses me, and jabber about that which holds my interest most: *nix.
On-prem Kubernetes, Part 1.75
Tags kubernetes, linux, networking, openbsd, virtualization
Posts in this series:
Project Goals and Description: Background info and goals
Preparing the installers (this post): pxeboot configs
Installing the Xen hosts: installing Debian/Xen
Installing the K8s VMs: installing the k8s domUs
Initializing the Control Plane: Bootstrapping a bare-bones HA Kubernetes Cluster
Installation/Configuration of Calico/MetalLB/ingress-nginx: Installing the CNI/Network Infrastructure
Installation/Configuration of LVM-CSI, S3-CSI, and Kadalu (GlusterFS): Installing the CSIs for Persistent Volumes
Installation/Configuration of cert-manager: Installing/Configuring cert-manager
Automating the boring bits: Installing/Configuring ArgoCD and GitOps Concepts
Authentication considerations: Installing/Configuring Authelia/Vault and LDAP/OAuth Integrations
Authentication configurations: Securing Applications with Authelia
Staying up to date: Keeping your cluster up-to-date
Github for example configuration files: rocket357/on-prem-kubernetes
Overview
It isn't Saturday, yet, so I haven't picked up the refurbished desktop machines I plan on using for my HA homelab kubernetes cluster. But in the spirit of being ready the moment I get home with them, I'm doing prep work today.
I'll be honest, I *hate* installing stuff, at least more than once. Getting a pxe boot environment setup is something I've written about before (here and the update here), so I think I'll set something like that up again, with Alpine-specific auto-installation scripts for the Xen hosts, and Debian-specific auto-installation configs for the VMs.
I already have an #OpenBSD relayd/httpd configuration handling access control from the internet and providing various files to my internal network, so adding the configurations to that (as opposed to relying just on tftp for files) is something I'm going to change from my previous posts. tftpd is easy to use, but it's fairly slow, so for full OS images it is better to serve those via httpd in the pxe config (UPDATE: I ended up going with tftp-only, as I wanted to keep boot stuff separate from my public httpd server).
The workflow for a fresh boot, then, will be:
host bios -> configure for pxe boot -> dhcp a network address -> dhcpd assign the host's pre-assigned address (and provide the pxeboot nextserver, which points to tftpd) -> host requests the configuration for their mac address -> tftp returns the configuration, which links to files over httpd -> host pull the required iso/scripts/etc... and lastly -> magic.
Maybe not magic, but it sure feels that way when you have to perform the same installation 19 times and pxe handles it all for you.
OpenBSD pxeboot vs Syslinux pxelinux vs debian netboot
I could go with OpenBSD's pxeboot binaries here, but as I pointed out in my previous pxeboot posts, OpenBSD's pxeboot is really amazing for pxebooting OpenBSD, but my Xen hosts will be running Linux so I'll need to go with syslinux again for this configuration (UPDATE: Debian netboot has essentially preconfigured pxelinux bits that work really, really well, so I ended up using that instead of rolling my own. The rest of this paragraph was just me experimenting with manually setting up pxeboot and can be ignored). I've downloaded the latest syslinux on my tftp host, untar'd it, and copied the memdisk, vesamenu.c32, and lpxelinux.0 files from appropriate places into the tftp root (note: lpxelinux.0 is required instead of pxelinux.0 since I'm going to be using http paths). Also note that I'm going to use the efi64 vesamenu.c32, which is a departure from my previous pxeboot posts, since a lot has changed since those posts and I'd rather not turn off efi on my Xen hosts unless I absolutely have to...so keep in mind this configuration may have to change this Saturday =) Next up, I created /tftp/pxelinux.cfg to hold all of the mac-specific configs. While here, I figure I might as well create /tftp/pxelinux.cfg/default with some useful utilities (basically the same thing as my previous posts). For completeness, here is /tftp/pxelinux.cfg/default:
I don't think I'm going to worry about running the livecds section per my previous posts, so I'll just stick to utilities and install images for now. Speaking of install images, I'll grab alpine latest's (3.19 as of this writing), both the iso and netboot bits (I may go either way, not sure yet) and put them in an appropriate place for httpd to serve them. For completeness, these go in /var/www/htdocs/images and /var/www/htdocs/boot/alpine/3.19.0, respectively. Same thing for Debian (with the appropriate boot/debian/12.2.0-amd64 path, of course).
Managing Expectations
Alright, at this point I think I can start writing the pxeboot config files. The first issue that comes to mind is that I don't have MAC addresses for the physical hosts yet. This will need to be addressed once I get the hosts home on Saturday, but for now I'll write out a simple default-xen boot file. Since the VMs will have Xen-supplied MAC addresses, I can write those now. (Those will begin with 00:16:3e, and I will randomize the remaining portion of the MAC).
Now you can see from the above default config file that I set "local boot" as the default. I'll do the same for the host-specific config files. After a brief timeout, it will proceed to boot from the local disk if no keyboard input is provided. I did this because during the OS installs for the hosts, I'm going to plug in a KVM to them to go through the BIOS settings anyways (pxeboot typically wouldn't be enabled, so I have to do this regardless for first boot), so on first boot up I'll bypass the local boot so pxeboot takes over and the OS gets installed. I don't suspect I'll get it perfect on first try, so I'm setting it this way for tinkering until I get it right.
A Xen host boot config might look like this:
It does without saying, but this is currently untested =) I'll provide updates/confirmation once I have the hosts ready to go.
The VM configs will look similar, except they'll contain pointers to the Debian configs, as well as some of the Debian autoinstall bits, which I'll iron out probably during setup (updates to follow, of course).
Until I get the hosts and have a chance to test these bits, I'll leave this for now. I'll update once I have the configurations ironed out (and I'll add the auto-install bits once I have them working properly).
Cheers!
Project Goals and Description: Background info and goals
Preparing the installers (this post): pxeboot configs
Installing the Xen hosts: installing Debian/Xen
Installing the K8s VMs: installing the k8s domUs
Initializing the Control Plane: Bootstrapping a bare-bones HA Kubernetes Cluster
Installation/Configuration of Calico/MetalLB/ingress-nginx: Installing the CNI/Network Infrastructure
Installation/Configuration of LVM-CSI, S3-CSI, and Kadalu (GlusterFS): Installing the CSIs for Persistent Volumes
Installation/Configuration of cert-manager: Installing/Configuring cert-manager
Automating the boring bits: Installing/Configuring ArgoCD and GitOps Concepts
Authentication considerations: Installing/Configuring Authelia/Vault and LDAP/OAuth Integrations
Authentication configurations: Securing Applications with Authelia
Staying up to date: Keeping your cluster up-to-date
Github for example configuration files: rocket357/on-prem-kubernetes
Overview
It isn't Saturday, yet, so I haven't picked up the refurbished desktop machines I plan on using for my HA homelab kubernetes cluster. But in the spirit of being ready the moment I get home with them, I'm doing prep work today.
I'll be honest, I *hate* installing stuff, at least more than once. Getting a pxe boot environment setup is something I've written about before (here and the update here), so I think I'll set something like that up again, with Alpine-specific auto-installation scripts for the Xen hosts, and Debian-specific auto-installation configs for the VMs.
I already have an #OpenBSD relayd/httpd configuration handling access control from the internet and providing various files to my internal network, so adding the configurations to that (as opposed to relying just on tftp for files) is something I'm going to change from my previous posts. tftpd is easy to use, but it's fairly slow, so for full OS images it is better to serve those via httpd in the pxe config (UPDATE: I ended up going with tftp-only, as I wanted to keep boot stuff separate from my public httpd server).
The workflow for a fresh boot, then, will be:
host bios -> configure for pxe boot -> dhcp a network address -> dhcpd assign the host's pre-assigned address (and provide the pxeboot nextserver, which points to tftpd) -> host requests the configuration for their mac address -> tftp returns the configuration, which links to files over httpd -> host pull the required iso/scripts/etc... and lastly -> magic.
Maybe not magic, but it sure feels that way when you have to perform the same installation 19 times and pxe handles it all for you.
OpenBSD pxeboot vs Syslinux pxelinux vs debian netboot
I could go with OpenBSD's pxeboot binaries here, but as I pointed out in my previous pxeboot posts, OpenBSD's pxeboot is really amazing for pxebooting OpenBSD, but my Xen hosts will be running Linux so I'll need to go with syslinux again for this configuration (UPDATE: Debian netboot has essentially preconfigured pxelinux bits that work really, really well, so I ended up using that instead of rolling my own. The rest of this paragraph was just me experimenting with manually setting up pxeboot and can be ignored). I've downloaded the latest syslinux on my tftp host, untar'd it, and copied the memdisk, vesamenu.c32, and lpxelinux.0 files from appropriate places into the tftp root (note: lpxelinux.0 is required instead of pxelinux.0 since I'm going to be using http paths). Also note that I'm going to use the efi64 vesamenu.c32, which is a departure from my previous pxeboot posts, since a lot has changed since those posts and I'd rather not turn off efi on my Xen hosts unless I absolutely have to...so keep in mind this configuration may have to change this Saturday =) Next up, I created /tftp/pxelinux.cfg to hold all of the mac-specific configs. While here, I figure I might as well create /tftp/pxelinux.cfg/default with some useful utilities (basically the same thing as my previous posts). For completeness, here is /tftp/pxelinux.cfg/default:
Code:
DEFAULT vesamenu.c32
prompt 0
MENU title PXE Boot Menu
MENU AUTOBOOT Starting Local System in # seconds
LABEL local_disk
MENU label ^Boot Local Hard drive
MENU default
LOCALBOOT -1
timeout 200
LABEL utilities
MENU label ^Utilities
KERNEL vesamenu.c32
APPEND pxelinux.cfg/utilities.menu
LABEL installers
MENU label ^Install Images
KERNEL vesamenu.c32
APPEND pxelinux.cfg/installers.menu
Managing Expectations
Alright, at this point I think I can start writing the pxeboot config files. The first issue that comes to mind is that I don't have MAC addresses for the physical hosts yet. This will need to be addressed once I get the hosts home on Saturday, but for now I'll write out a simple default-xen boot file. Since the VMs will have Xen-supplied MAC addresses, I can write those now. (Those will begin with 00:16:3e, and I will randomize the remaining portion of the MAC).
Now you can see from the above default config file that I set "local boot" as the default. I'll do the same for the host-specific config files. After a brief timeout, it will proceed to boot from the local disk if no keyboard input is provided. I did this because during the OS installs for the hosts, I'm going to plug in a KVM to them to go through the BIOS settings anyways (pxeboot typically wouldn't be enabled, so I have to do this regardless for first boot), so on first boot up I'll bypass the local boot so pxeboot takes over and the OS gets installed. I don't suspect I'll get it perfect on first try, so I'm setting it this way for tinkering until I get it right.
A Xen host boot config might look like this:
Code:
MENU title Xenboot
MENU AUTOBOOT Starting Local System in # seconds
LABEL local_disk
MENU label ^Boot Local Hard drive
MENU default
LOCALBOOT -1
timeout 200
LABEL xenboot
MENU label ^Alpine/Xen install
KERNEL memdisk
APPEND http://$HTTPD_IP/images/alpine-standard-3.19.0-x86_64.iso
The VM configs will look similar, except they'll contain pointers to the Debian configs, as well as some of the Debian autoinstall bits, which I'll iron out probably during setup (updates to follow, of course).
Until I get the hosts and have a chance to test these bits, I'll leave this for now. I'll update once I have the configurations ironed out (and I'll add the auto-install bits once I have them working properly).
Cheers!
Total Comments 0
