Hi. I'm jon.404, a Unix/Linux/Database/Openstack/Kubernetes Administrator, AWS/GCP/Azure Engineer, mathematics enthusiast, and amateur philosopher. This is where I rant about that which upsets me, laugh about that which amuses me, and jabber about that which holds my interest most: *nix.
On-prem kubernetes, Part 1.99999
Tags kubernetes, linux, networking, openbsd, virtualization
Posts in this series:
Project Goals and Description: Background info and goals
Preparing the installers: pxeboot configs
Installing the Xen Hosts (this post): installing Debian/Xen
Installing the K8s VMs: installing the k8s domUs
Initializing the Control Plane: Bootstrapping a bare-bones HA Kubernetes Cluster
Installation/Configuration of Calico/MetalLB/ingress-nginx: Installing the CNI/Network Infrastructure
Installation/Configuration of LVM-CSI, S3-CSI, and Kadalu (GlusterFS): Installing the CSIs for Persistent Volumes
Installation/Configuration of cert-manager: Installing/Configuring cert-manager
Automating the boring bits: Installing/Configuring ArgoCD and GitOps Concepts
Authentication considerations: Installing/Configuring Authelia/Vault and LDAP/OAuth Integrations
Authentication configurations: Securing Applications with Authelia
Staying up to date: Keeping your cluster up-to-date
Github for example configuration files: rocket357/on-prem-kubernetes
Overview
Saturday has come and gone, and after spending a considerable portion of the day hammering away at the pxeboot configurations, I've made a few changes:
The Switch to Debian
In the interest of not getting mired down in rabbit holes, I've switched to Xen/Debian. The reason has nothing to do with Alpine's usability or stability or the like. In fact, it's actually somewhat dumb. pxebooting iso images has always been a bit of a hit and miss, with some isos requiring files off the iso post-boot. That sounds simple, but in fact it is quite a fuster cluck. During cdrom/usb usage, the boot bits are read off of the media and the machine boots up with it. At some point during boot, the cdrom/usb image is mounted, and additional files are read off of the image. Booting continues normally. With pxe, however, there is no physical medium, so the image is transferred *in whole* to the client's memory, then the boot bits are executed. If the iso is constructed with this in mind (i.e. no additional files are required from the iso), then you're off to the races. However, many modern isos use squashfs to compress the non-boot bits of the image to save space, so the squashfs files need to be read from the image to continue booting. This breaks the "normal" workflow of pxebooting and causes the boot process to halt unceremoniously because the iso doesn't expect the files to be available in memory...it expects them to be available on a physical medium.
By sheer luck, the developers at Debian have a working solution to this issue...a mini.iso that is constructed with pxebooting in mind. It literally boots into a "micro OS" that then downloads the netinst bits, which in turn download the remaining bits to actually perform the install. Think of it as a "netinst for netinst". It's small enough (~60MB...coming from OpenBSD that's insanely bloated, but...Linux) that pxebooting doesn't require any additional file reads and the installer continues seamlessly (assuming you have internet and can download the remainder). As promised, I'll update the older blog posts with that information. Note: Alpine probably has a similar iso available, but given a few systemd/kubernetes related reasons (node-problem-detector is something I'll want to run in the final k8s cluster), I decided to switch to Debian regardless.
HA Requirements
There's one small detail that my previous posts left out that I need to discuss currently. Building a single control plane k8s cluster is fairly straightforward. You install the kubelet and configure everything (kubeadm or whatever) and then you simply add workers to the cluster with the provided join command. For an HA control plane, however, you need a load balancer that can distribute requests among the control plane nodes. This has to exist and be usable (DNS, etc...) *BEFORE* you install the control plane. Being on-prem, this means we'll need to utilize a software loadbalancer. There are many out there, but possibly the most widely used (if google search counts mean anything) is haproxy/keepalived. Setting them up was straightforward and given the chicken and egg problem of haproxy/keepalived being up and usable before the control plane, you can't simply run a helm install or kubectl apply after the fact (man that would be nice, though). The software loadbalancer has to exist prior to the control plane (i.e. before the kubernetes cluster exists). There are different ways of dealing with this issue in a Xen environment...you can run the loadbalancer in dom0, or in your virtualized control plane domUs (or even in a separate domU...you have choices here). I'm not going to tell you how to live, but I'd like to keep dom0 as simple as possible and I'd like to keep the number of VMs to a minimum, so I'll be installing haproxy/keepalived in the control plane domUs. (Update: Due to how I'm running the kube-apiserver on the domUs, this cause port conflicts. I could move haproxy to another port, but in the interest of running a "clean" install/configuration, I decided to move haproxy/keepalived to the dom0's).
Testing the Theories
For the purposes of installing Debian/Xen, I booted xen1 off the mini.iso from pxe and manually configured everything to ensure it all works. This step goes against everything that the pxe automation stands for, but once I've gotten everything ironed out, I'm just going to pxeboot re-install xen1 from the automated install process and (hopefully) only have had to do the manual steps once (and then xen2/xen3 will go seamlessly up into operation...barring any stupidity on my side, of course). Theory and practice, as they say! After discovering the remaining bits that debian has in place for pxebooting, I promptly ran headlong into a kernel issue, scratched my head for a few minutes on it, read an announcement on Reddit that the kernel issue was resolved, and proceeded to continue about my merry way. Secure boot on these HP devices is being dorky and not letting me boot Xen, so I've disabled secure boot (again, after the installer seems to not care, but the running system does care) and I'm back to ground zero. More on pxe automation shenanigans in a moment...
TFTP vs HTTPD
I also mentioned in #2 above that I am going to keep everything in tftp instead of httpd. There are a few reasons for this, and some of it is general laziness on my part. tftp works, of course, but is slow compared to httpd transfers (usually...it uses udp instead of tcp and there are some gotchas to it). But httpd in my configuration is listening on 127.0.0.1 with relayd in front redirecting traffic (why not use relayd as my software loadbalancer, you ask? Read on!). I spent about 20 minutes looking at options and trying to decide how to architect the ACLs (relayd is open to the internet, so it controls access to httpd). In the end, I decided the best way to keep public and private separate is to just...keep them separate. Thus all of my boot configs/images/etc... will reside in tftp and httpd will only do httpd things. If 20+ years in IT/CloudOps has taught me anything, it's that *the infrastructure is not the end goal*. The infrastructure exists to support the applications that users utilize. (Likewise, I'm not married to Alpine for Alpine's sake. I'd already run in to some issues running kubernetes workloads on Alpine (node-problem-detector specifically comes to mind as it relies on systemd for watching logs), and I was already going to have a Debian pxeboot environment set up...why not just run it all on Debian, then, and kill two birds with one stone?).
Xen Install
The Xen install (after booting from my modified debian netboot bits) is incredibly plagiarized. I legit followed the Xen Project Beginner's Guide, which utilized Debian. I'd rather not have to keep this blog post constantly updated, so I'm just going to point out how I've automated the above and leave the link available for specifics. In terms of automation, preseed FTW.
Speaking of which, I'll be posting example code and configurations on github (Link to follow once I've gotten that ironed out).
Alright, time for some pxeboot preseed shenanigans!
First up, as if the kernel issue and the efi tomfoolery wasn't enough, getting a Debian preseed file *just right* is an art in and of itself. The first thing I noticed is that the partitioner wasn't set correctly (there's a bug where max sizes aren't honored, it seems), so the entire disk is used by dom0 unless you have a final "max size" partition. This needs to change in my preseed, so I setup a 20G dom0 root lv, a 100G controlplane lv, a 100G worker lv, and finally a "max" size pvcs lv (for kubernetes persistent volume claims). Also, during the (many) boots on pxe, I ran in to various other issues I've since resolved (asking for default packages, asking about popularity-contest, etc...). Once I've updated the partitioner details and installed once again, I have a working Xen/Debian dom0.
Now I just roll out the same to xen2 and xen3, and I have my hypervisors ready to go.
Next up (perhaps tonight?) the kubernetes installations!
Project Goals and Description: Background info and goals
Preparing the installers: pxeboot configs
Installing the Xen Hosts (this post): installing Debian/Xen
Installing the K8s VMs: installing the k8s domUs
Initializing the Control Plane: Bootstrapping a bare-bones HA Kubernetes Cluster
Installation/Configuration of Calico/MetalLB/ingress-nginx: Installing the CNI/Network Infrastructure
Installation/Configuration of LVM-CSI, S3-CSI, and Kadalu (GlusterFS): Installing the CSIs for Persistent Volumes
Installation/Configuration of cert-manager: Installing/Configuring cert-manager
Automating the boring bits: Installing/Configuring ArgoCD and GitOps Concepts
Authentication considerations: Installing/Configuring Authelia/Vault and LDAP/OAuth Integrations
Authentication configurations: Securing Applications with Authelia
Staying up to date: Keeping your cluster up-to-date
Github for example configuration files: rocket357/on-prem-kubernetes
Overview
Saturday has come and gone, and after spending a considerable portion of the day hammering away at the pxeboot configurations, I've made a few changes:
- Xen with Debian Dom0
- software loadbalancing (HA control plane requirement)
- tftp for isos
The Switch to Debian
In the interest of not getting mired down in rabbit holes, I've switched to Xen/Debian. The reason has nothing to do with Alpine's usability or stability or the like. In fact, it's actually somewhat dumb. pxebooting iso images has always been a bit of a hit and miss, with some isos requiring files off the iso post-boot. That sounds simple, but in fact it is quite a fuster cluck. During cdrom/usb usage, the boot bits are read off of the media and the machine boots up with it. At some point during boot, the cdrom/usb image is mounted, and additional files are read off of the image. Booting continues normally. With pxe, however, there is no physical medium, so the image is transferred *in whole* to the client's memory, then the boot bits are executed. If the iso is constructed with this in mind (i.e. no additional files are required from the iso), then you're off to the races. However, many modern isos use squashfs to compress the non-boot bits of the image to save space, so the squashfs files need to be read from the image to continue booting. This breaks the "normal" workflow of pxebooting and causes the boot process to halt unceremoniously because the iso doesn't expect the files to be available in memory...it expects them to be available on a physical medium.
By sheer luck, the developers at Debian have a working solution to this issue...a mini.iso that is constructed with pxebooting in mind. It literally boots into a "micro OS" that then downloads the netinst bits, which in turn download the remaining bits to actually perform the install. Think of it as a "netinst for netinst". It's small enough (~60MB...coming from OpenBSD that's insanely bloated, but...Linux) that pxebooting doesn't require any additional file reads and the installer continues seamlessly (assuming you have internet and can download the remainder). As promised, I'll update the older blog posts with that information. Note: Alpine probably has a similar iso available, but given a few systemd/kubernetes related reasons (node-problem-detector is something I'll want to run in the final k8s cluster), I decided to switch to Debian regardless.
HA Requirements
There's one small detail that my previous posts left out that I need to discuss currently. Building a single control plane k8s cluster is fairly straightforward. You install the kubelet and configure everything (kubeadm or whatever) and then you simply add workers to the cluster with the provided join command. For an HA control plane, however, you need a load balancer that can distribute requests among the control plane nodes. This has to exist and be usable (DNS, etc...) *BEFORE* you install the control plane. Being on-prem, this means we'll need to utilize a software loadbalancer. There are many out there, but possibly the most widely used (if google search counts mean anything) is haproxy/keepalived. Setting them up was straightforward and given the chicken and egg problem of haproxy/keepalived being up and usable before the control plane, you can't simply run a helm install or kubectl apply after the fact (man that would be nice, though). The software loadbalancer has to exist prior to the control plane (i.e. before the kubernetes cluster exists). There are different ways of dealing with this issue in a Xen environment...you can run the loadbalancer in dom0, or in your virtualized control plane domUs (or even in a separate domU...you have choices here). I'm not going to tell you how to live, but I'd like to keep dom0 as simple as possible and I'd like to keep the number of VMs to a minimum, so I'll be installing haproxy/keepalived in the control plane domUs. (Update: Due to how I'm running the kube-apiserver on the domUs, this cause port conflicts. I could move haproxy to another port, but in the interest of running a "clean" install/configuration, I decided to move haproxy/keepalived to the dom0's).
Testing the Theories
For the purposes of installing Debian/Xen, I booted xen1 off the mini.iso from pxe and manually configured everything to ensure it all works. This step goes against everything that the pxe automation stands for, but once I've gotten everything ironed out, I'm just going to pxeboot re-install xen1 from the automated install process and (hopefully) only have had to do the manual steps once (and then xen2/xen3 will go seamlessly up into operation...barring any stupidity on my side, of course). Theory and practice, as they say! After discovering the remaining bits that debian has in place for pxebooting, I promptly ran headlong into a kernel issue, scratched my head for a few minutes on it, read an announcement on Reddit that the kernel issue was resolved, and proceeded to continue about my merry way. Secure boot on these HP devices is being dorky and not letting me boot Xen, so I've disabled secure boot (again, after the installer seems to not care, but the running system does care) and I'm back to ground zero. More on pxe automation shenanigans in a moment...
TFTP vs HTTPD
I also mentioned in #2 above that I am going to keep everything in tftp instead of httpd. There are a few reasons for this, and some of it is general laziness on my part. tftp works, of course, but is slow compared to httpd transfers (usually...it uses udp instead of tcp and there are some gotchas to it). But httpd in my configuration is listening on 127.0.0.1 with relayd in front redirecting traffic (why not use relayd as my software loadbalancer, you ask? Read on!). I spent about 20 minutes looking at options and trying to decide how to architect the ACLs (relayd is open to the internet, so it controls access to httpd). In the end, I decided the best way to keep public and private separate is to just...keep them separate. Thus all of my boot configs/images/etc... will reside in tftp and httpd will only do httpd things. If 20+ years in IT/CloudOps has taught me anything, it's that *the infrastructure is not the end goal*. The infrastructure exists to support the applications that users utilize. (Likewise, I'm not married to Alpine for Alpine's sake. I'd already run in to some issues running kubernetes workloads on Alpine (node-problem-detector specifically comes to mind as it relies on systemd for watching logs), and I was already going to have a Debian pxeboot environment set up...why not just run it all on Debian, then, and kill two birds with one stone?).
Xen Install
The Xen install (after booting from my modified debian netboot bits) is incredibly plagiarized. I legit followed the Xen Project Beginner's Guide, which utilized Debian. I'd rather not have to keep this blog post constantly updated, so I'm just going to point out how I've automated the above and leave the link available for specifics. In terms of automation, preseed FTW.
Speaking of which, I'll be posting example code and configurations on github (Link to follow once I've gotten that ironed out).
Alright, time for some pxeboot preseed shenanigans!
First up, as if the kernel issue and the efi tomfoolery wasn't enough, getting a Debian preseed file *just right* is an art in and of itself. The first thing I noticed is that the partitioner wasn't set correctly (there's a bug where max sizes aren't honored, it seems), so the entire disk is used by dom0 unless you have a final "max size" partition. This needs to change in my preseed, so I setup a 20G dom0 root lv, a 100G controlplane lv, a 100G worker lv, and finally a "max" size pvcs lv (for kubernetes persistent volume claims). Also, during the (many) boots on pxe, I ran in to various other issues I've since resolved (asking for default packages, asking about popularity-contest, etc...). Once I've updated the partitioner details and installed once again, I have a working Xen/Debian dom0.
Now I just roll out the same to xen2 and xen3, and I have my hypervisors ready to go.
Next up (perhaps tonight?) the kubernetes installations!
Total Comments 0
