Hi. I'm jon.404, a Unix/Linux/Database/Openstack/Kubernetes Administrator, AWS/GCP/Azure Engineer, mathematics enthusiast, and amateur philosopher. This is where I rant about that which upsets me, laugh about that which amuses me, and jabber about that which holds my interest most: *nix.
On-premise kubernetes, Part 1.5
Tags kubernetes, linux, networking, openbsd, virtualization
Posts in this series:
Project Goals and Description (this post): Background info and goals
Preparing the installers: pxeboot configs
Installing the Xen hosts: installing Debian/Xen
Installing the K8s VMs: installing the k8s domUs
Initializing the Control Plane: Bootstrapping a bare-bones HA Kubernetes Cluster
Installation/Configuration of Calico/MetalLB/ingress-nginx: Installing the CNI/Network Infrastructure
Installation/Configuration of LVM-CSI, S3-CSI, and Kadalu (GlusterFS): Installing the CSIs for Persistent Volumes
Installation/Configuration of cert-manager: Installing/Configuring cert-manager
Automating the boring bits: Installing/Configuring ArgoCD and GitOps Concepts
Authentication considerations: Installing/Configuring Authelia/Vault and LDAP/OAuth Integrations
Authentication configurations: Securing Applications with Authelia
Staying up to date: Keeping your cluster up-to-date
Github for example configuration files: rocket357/on-prem-kubernetes
Overview
I've been playing around with an old gaming laptop that has quite a bit of horsepower (hyperthreaded i7/32GB RAM) as a xen hypervisor for running a minimalist kubernetes cluster (single control plane node, three workers, flannel CNI, LVM CSI...this was the "Part 1" alluded to in the title, but for which there exists no blog post).
I liked the POC so much that I decided to buy three refurbished desktop machines...I have a friend that runs a refurbishing company and could get me some deep discounts...to expand out to an HA configuration, but also for testing some more complex integrations into my BGP/OSPF home network backbone. Calico is a kubernetes CNI that does BGP meshing (no encapsulation, so it's supposed to be near-wirespeed for pod networking...we'll see on that) which is a nice step up from Flannel (Canal is a nice middle ground, of course, but it uses flannel for the inter-pod communications, so back to encapsulation).
So this blog post is going to cover some of the background into the complexities of what I'm trying to accomplish with my home network, as well as some of the challenges for non-cloud home kubernetes cluster configurations. Then on Saturday I'll be driving down to my buddy's house to pick up the refurbs, and probably over the next few days will publish part 2 of this blog series.
Use the Right Tool for the Job
If you read much of my previous blog rants, you'll know that I'm an avid OpenBSD user. I use OpenBSD everywhere I can, from "on-premise" security appliances (firewalls, proxies, and the like) to virtual machines out in the cloud (mostly for VPNs, bgp, and ospf). But if I'm being honest, while OpenBSD does absolutely kick a$$ in the security and networking departments, it is still quite in its infancy with regards to virtualization. It performs nicely as a virtual guest, of course, but as a virtualization host it is lagging behind what I would consider "modern". (I really, *really* like vmd's secure approach to virtualization, don't get me wrong...but single core, no graphics VMs and complete lack of containerization in OpenBSD leaves a lot to be desired...but you bet your bottom dollar if that changes in the future I'll be switching to OpenBSD vmd hosts in a heartbeat).
So the main takeaway here is to use the right tool for the job. I play PC games with my wife and kids, and for that I use a Windows machine (I know...bleh...I ran Wine on Linux for a long time and got tired of fixing every game after any upgrades to the games, wine, the OS I was using, etc...). I have security and networking devices, and for those I use OpenBSD. I want to build a kubernetes cluster, and for that I use Linux.
To HA or not to HA
The eagle-eyed among you will notice that I said I bought three machines for this HA kubernetes cluster, and HA in kubernetes requires a minimum of three control plane nodes. Also, control plane nodes do not run workloads...so how am I supposed to run a full cluster on just three devices? I could remove the workload taint on the control plane devices, and thus run workloads on them, or I could virtualize the control plane hosts and create workers alongside them. I like the idea of the control plane hosts being dedicated ("dedicated" to being control plane workers and not running workloads, not "dedicated" as in hardware), so I'm going to go with Xen for the underlying hypervisor. There are many options here, of course, but I've run Alpine/Xen before and it works quite well. This could be accomplished in VirtualBox, KVM, etc...in theory, but in practice I'm going to stick with Alpine/Xen (UPDATE: I ended up going with Debian/Xen as pxeboot/autoinstall was a bit easier for me to iron out).
Networking Fun
The next big consideration is the networking aspect of my home network and kubernetes. I have an OSPF backbone running across vpn links to a few sites (i.e. two regional endpoints running in a cloud provider that interconnect with a few services as well as allow me to route specific traffic across the VPN links) that is in turn bgp-peered with additional networks (friends and such) across additional VPN links. The BGP peers feed routes into my ospf backbone, and those are redistributed across my network to all of the other BGP peers (I'm a poor-man's transit AS, I guess). The end goal here is to have a dynamically updated route table for services that my friends run (that I use from time to time) and for services that I run (that my friends use from time to time). Some of the services are "region-specific", such as OpenWebRX+ and other radiowave services (adsb and such) that can't be realistically "copied" to another location since they rely on regional radiowave listening and the like. Granted, most services don't require that level of regional "stickiness" and can be run essentially anywhere (I'm in discussion with one of my friends up north in setting up a dashboard so I can run workloads on his cluster and he can run workloads on mine...but we haven't ironed out all of the details on that yet). I could, for example, run mealie on a friend's cluster, and perhaps someday we'll get to "regional failover" capabilities (yikes...though I could see postgres-operator running standby database clusters across VPN links...SMH), but for now we just run whatever we're interested in and offer that up to other friends to use.
Currently I've found that MetalLB does a fantastic job with BGP peering (by default it does arp-based "load balancing", but in BGP mode you can have redundant links to services), so I simply peer the kubernetes MetalLB Daemonset with my home firewall (running OpenBGPD on OpenBSD) and those routes get injected to the OSPF routes, redistributed, and my friends have them essentially immediately. So, for instance, if I do a rolling update of a deployment in kubernetes, and pods/services/etc... move around to different hosts, MetalLB pushes BGP updates to route those requests to the new hosts. My friends, then, get updates on routes (summarized, of course, but they get them in near real time) without me having to do any manual lifting. This also gives me the ability to jump on irc and laugh at one of my friends if they are having network issues...but that's an entirely different story.
One thought is that if flannel with MetalLB already does BGP peering, why switch to Calico? Aside from the fact that Calico has network policy support (that flannel lacks), and Calico does BGP meshing within the kubernetes cluster, I'm interested in pushing the bounds to see if Calico can extend further into the outside network with some of the other features it supports (SCTP, for instance). I'll post more on this as I figure it out in the following days. (Also Note: MetalLB is not going to be going away when I switch to Calico).
Probably the final consideration at this point is TLS, which I'm consuming via LetsEncrypt. I have a single wildcard cert on my firewall (used in relayd to route packets to the appropriate internal hosts for services...OpenBSD's acme-client doesn't handle wildcard certs since that requires DNS challenges, so I had to switch to acme.sh for the wildcard cert) and then each "service host" (kubernetes or otherwise) is responsible for interfacing with LetsEncrypt to grab service-specific certificates. I hated the idea of having a list of service urls directly in a public-facing certificate, so a wildcard makes it a bit less worrisome. The end result is that whether you hit a service from the public internet or from an internal route, you get a secure, verified connection (ironically, OpenBSD needed the LetsEncrypt intermediate certs added to /etc/ssl/cert.pem, but I digress).
Next Steps
That pretty much lays out the groundwork for the implementation and what to expect in future posts. I'll be back in a few days with the Xen/Alpine/VM/kubernetes installations and let you guys know how that goes.
Cheers!
Project Goals and Description (this post): Background info and goals
Preparing the installers: pxeboot configs
Installing the Xen hosts: installing Debian/Xen
Installing the K8s VMs: installing the k8s domUs
Initializing the Control Plane: Bootstrapping a bare-bones HA Kubernetes Cluster
Installation/Configuration of Calico/MetalLB/ingress-nginx: Installing the CNI/Network Infrastructure
Installation/Configuration of LVM-CSI, S3-CSI, and Kadalu (GlusterFS): Installing the CSIs for Persistent Volumes
Installation/Configuration of cert-manager: Installing/Configuring cert-manager
Automating the boring bits: Installing/Configuring ArgoCD and GitOps Concepts
Authentication considerations: Installing/Configuring Authelia/Vault and LDAP/OAuth Integrations
Authentication configurations: Securing Applications with Authelia
Staying up to date: Keeping your cluster up-to-date
Github for example configuration files: rocket357/on-prem-kubernetes
Overview
I've been playing around with an old gaming laptop that has quite a bit of horsepower (hyperthreaded i7/32GB RAM) as a xen hypervisor for running a minimalist kubernetes cluster (single control plane node, three workers, flannel CNI, LVM CSI...this was the "Part 1" alluded to in the title, but for which there exists no blog post).
I liked the POC so much that I decided to buy three refurbished desktop machines...I have a friend that runs a refurbishing company and could get me some deep discounts...to expand out to an HA configuration, but also for testing some more complex integrations into my BGP/OSPF home network backbone. Calico is a kubernetes CNI that does BGP meshing (no encapsulation, so it's supposed to be near-wirespeed for pod networking...we'll see on that) which is a nice step up from Flannel (Canal is a nice middle ground, of course, but it uses flannel for the inter-pod communications, so back to encapsulation).
So this blog post is going to cover some of the background into the complexities of what I'm trying to accomplish with my home network, as well as some of the challenges for non-cloud home kubernetes cluster configurations. Then on Saturday I'll be driving down to my buddy's house to pick up the refurbs, and probably over the next few days will publish part 2 of this blog series.
Use the Right Tool for the Job
If you read much of my previous blog rants, you'll know that I'm an avid OpenBSD user. I use OpenBSD everywhere I can, from "on-premise" security appliances (firewalls, proxies, and the like) to virtual machines out in the cloud (mostly for VPNs, bgp, and ospf). But if I'm being honest, while OpenBSD does absolutely kick a$$ in the security and networking departments, it is still quite in its infancy with regards to virtualization. It performs nicely as a virtual guest, of course, but as a virtualization host it is lagging behind what I would consider "modern". (I really, *really* like vmd's secure approach to virtualization, don't get me wrong...but single core, no graphics VMs and complete lack of containerization in OpenBSD leaves a lot to be desired...but you bet your bottom dollar if that changes in the future I'll be switching to OpenBSD vmd hosts in a heartbeat).
So the main takeaway here is to use the right tool for the job. I play PC games with my wife and kids, and for that I use a Windows machine (I know...bleh...I ran Wine on Linux for a long time and got tired of fixing every game after any upgrades to the games, wine, the OS I was using, etc...). I have security and networking devices, and for those I use OpenBSD. I want to build a kubernetes cluster, and for that I use Linux.
To HA or not to HA
The eagle-eyed among you will notice that I said I bought three machines for this HA kubernetes cluster, and HA in kubernetes requires a minimum of three control plane nodes. Also, control plane nodes do not run workloads...so how am I supposed to run a full cluster on just three devices? I could remove the workload taint on the control plane devices, and thus run workloads on them, or I could virtualize the control plane hosts and create workers alongside them. I like the idea of the control plane hosts being dedicated ("dedicated" to being control plane workers and not running workloads, not "dedicated" as in hardware), so I'm going to go with Xen for the underlying hypervisor. There are many options here, of course, but I've run Alpine/Xen before and it works quite well. This could be accomplished in VirtualBox, KVM, etc...in theory, but in practice I'm going to stick with Alpine/Xen (UPDATE: I ended up going with Debian/Xen as pxeboot/autoinstall was a bit easier for me to iron out).
Networking Fun
The next big consideration is the networking aspect of my home network and kubernetes. I have an OSPF backbone running across vpn links to a few sites (i.e. two regional endpoints running in a cloud provider that interconnect with a few services as well as allow me to route specific traffic across the VPN links) that is in turn bgp-peered with additional networks (friends and such) across additional VPN links. The BGP peers feed routes into my ospf backbone, and those are redistributed across my network to all of the other BGP peers (I'm a poor-man's transit AS, I guess). The end goal here is to have a dynamically updated route table for services that my friends run (that I use from time to time) and for services that I run (that my friends use from time to time). Some of the services are "region-specific", such as OpenWebRX+ and other radiowave services (adsb and such) that can't be realistically "copied" to another location since they rely on regional radiowave listening and the like. Granted, most services don't require that level of regional "stickiness" and can be run essentially anywhere (I'm in discussion with one of my friends up north in setting up a dashboard so I can run workloads on his cluster and he can run workloads on mine...but we haven't ironed out all of the details on that yet). I could, for example, run mealie on a friend's cluster, and perhaps someday we'll get to "regional failover" capabilities (yikes...though I could see postgres-operator running standby database clusters across VPN links...SMH), but for now we just run whatever we're interested in and offer that up to other friends to use.
Currently I've found that MetalLB does a fantastic job with BGP peering (by default it does arp-based "load balancing", but in BGP mode you can have redundant links to services), so I simply peer the kubernetes MetalLB Daemonset with my home firewall (running OpenBGPD on OpenBSD) and those routes get injected to the OSPF routes, redistributed, and my friends have them essentially immediately. So, for instance, if I do a rolling update of a deployment in kubernetes, and pods/services/etc... move around to different hosts, MetalLB pushes BGP updates to route those requests to the new hosts. My friends, then, get updates on routes (summarized, of course, but they get them in near real time) without me having to do any manual lifting. This also gives me the ability to jump on irc and laugh at one of my friends if they are having network issues...but that's an entirely different story.
One thought is that if flannel with MetalLB already does BGP peering, why switch to Calico? Aside from the fact that Calico has network policy support (that flannel lacks), and Calico does BGP meshing within the kubernetes cluster, I'm interested in pushing the bounds to see if Calico can extend further into the outside network with some of the other features it supports (SCTP, for instance). I'll post more on this as I figure it out in the following days. (Also Note: MetalLB is not going to be going away when I switch to Calico).
Probably the final consideration at this point is TLS, which I'm consuming via LetsEncrypt. I have a single wildcard cert on my firewall (used in relayd to route packets to the appropriate internal hosts for services...OpenBSD's acme-client doesn't handle wildcard certs since that requires DNS challenges, so I had to switch to acme.sh for the wildcard cert) and then each "service host" (kubernetes or otherwise) is responsible for interfacing with LetsEncrypt to grab service-specific certificates. I hated the idea of having a list of service urls directly in a public-facing certificate, so a wildcard makes it a bit less worrisome. The end result is that whether you hit a service from the public internet or from an internal route, you get a secure, verified connection (ironically, OpenBSD needed the LetsEncrypt intermediate certs added to /etc/ssl/cert.pem, but I digress).
Next Steps
That pretty much lays out the groundwork for the implementation and what to expect in future posts. I'll be back in a few days with the Xen/Alpine/VM/kubernetes installations and let you guys know how that goes.
Cheers!
Total Comments 0
