Hi. I'm jon.404, a Unix/Linux/Database/Openstack/Kubernetes Administrator, AWS/GCP/Azure Engineer, mathematics enthusiast, and amateur philosopher. This is where I rant about that which upsets me, laugh about that which amuses me, and jabber about that which holds my interest most: *nix.
On-prem kubernetes, Part 2
Tags kubernetes, linux, networking, openbsd, virtualization
Posts in this series:
Project Goals and Description: Background info and goals
Preparing the installers: pxeboot configs
Installing the Xen Hosts: installing Debian/Xen dom0
Installing the K8s VMs (this post): installing the k8s domUs
Initializing the Control Plane: Bootstrapping a bare-bones HA Kubernetes Cluster
Installation/Configuration of Calico/MetalLB/ingress-nginx: Installing the CNI/Network Infrastructure
Installation/Configuration of LVM-CSI, S3-CSI, and Kadalu (GlusterFS): Installing the CSIs for Persistent Volumes
Installation/Configuration of cert-manager: Installing/Configuring cert-manager
Automating the boring bits: Installing/Configuring ArgoCD and GitOps Concepts
Authentication considerations: Installing/Configuring Authelia/Vault and LDAP/OAuth Integrations
Authentication configurations: Securing Applications with Authelia
Staying up to date: Keeping your cluster up-to-date
Github for example configuration files: rocket357/on-prem-kubernetes
Overview
I've finalized the pxeboot configs for the Xen VMs, and I have to admit after going down the xenpvnetboot rabbit hole, I gave up and curl'd the debian vmlinuz and initrd.gz files for xen booting to each xen host so I could boot/install the VMs from them. Per this delete commit, xenpvnetboot wasn't sustainable (needed a complete rewrite), and honestly all it did was download the vmlinuz and initrd.gz files anyways...so I added that step to the xen-preseed config and now everything works. Sigh.
K8s Host Installs
The installation preseed for the k8s VMs is very similar to the xen host preseed files, save a few packages (i.e. no Xen/haproxy/keepalived, but add the kubernetes things) as well as having a different directory layout (no lvm, just a single root partition/filesystem), but a minor bonus is that the MAC addresses don't need to be set in the preseed (I set static Xensource MACs for the xenbr0 on the xen hosts for $REASONS relating to my dhcpd configuration), so I can just use a single preseed file for all k8s hosts. I also set up a DNS record for kube-apiserver.$MYTLD to point to haproxy's floating IP.
At this point honestly the hard work is complete, and it's just a matter of setting the configs correctly for each domU and booting them to a completely automated installation, then updating the xen domU config file (setting the bootloader line and commenting out the install-specific options), and booting it back up.
Next Steps
By now we should have three 2 vcpu/2 GB/100 GB kubernetes control plane domUs, and three 4 vcpu/16 GB/100 GB kubernetes worker domUs. The next post in this series will cover installing the kubernetes control plane and joining the workers to the cluster to form a barebones kubernetes cluster.
Cheers!
Project Goals and Description: Background info and goals
Preparing the installers: pxeboot configs
Installing the Xen Hosts: installing Debian/Xen dom0
Installing the K8s VMs (this post): installing the k8s domUs
Initializing the Control Plane: Bootstrapping a bare-bones HA Kubernetes Cluster
Installation/Configuration of Calico/MetalLB/ingress-nginx: Installing the CNI/Network Infrastructure
Installation/Configuration of LVM-CSI, S3-CSI, and Kadalu (GlusterFS): Installing the CSIs for Persistent Volumes
Installation/Configuration of cert-manager: Installing/Configuring cert-manager
Automating the boring bits: Installing/Configuring ArgoCD and GitOps Concepts
Authentication considerations: Installing/Configuring Authelia/Vault and LDAP/OAuth Integrations
Authentication configurations: Securing Applications with Authelia
Staying up to date: Keeping your cluster up-to-date
Github for example configuration files: rocket357/on-prem-kubernetes
Overview
I've finalized the pxeboot configs for the Xen VMs, and I have to admit after going down the xenpvnetboot rabbit hole, I gave up and curl'd the debian vmlinuz and initrd.gz files for xen booting to each xen host so I could boot/install the VMs from them. Per this delete commit, xenpvnetboot wasn't sustainable (needed a complete rewrite), and honestly all it did was download the vmlinuz and initrd.gz files anyways...so I added that step to the xen-preseed config and now everything works. Sigh.
K8s Host Installs
The installation preseed for the k8s VMs is very similar to the xen host preseed files, save a few packages (i.e. no Xen/haproxy/keepalived, but add the kubernetes things) as well as having a different directory layout (no lvm, just a single root partition/filesystem), but a minor bonus is that the MAC addresses don't need to be set in the preseed (I set static Xensource MACs for the xenbr0 on the xen hosts for $REASONS relating to my dhcpd configuration), so I can just use a single preseed file for all k8s hosts. I also set up a DNS record for kube-apiserver.$MYTLD to point to haproxy's floating IP.
At this point honestly the hard work is complete, and it's just a matter of setting the configs correctly for each domU and booting them to a completely automated installation, then updating the xen domU config file (setting the bootloader line and commenting out the install-specific options), and booting it back up.
Next Steps
By now we should have three 2 vcpu/2 GB/100 GB kubernetes control plane domUs, and three 4 vcpu/16 GB/100 GB kubernetes worker domUs. The next post in this series will cover installing the kubernetes control plane and joining the workers to the cluster to form a barebones kubernetes cluster.
Cheers!
Total Comments 0
