On-prem kubernetes, Part 6

Posts in this series:

1. Background info and goals
2. pxeboot configs
3. installing Debian/Xen dom0
4. installing the k8s domUs
5. Bootstrapping a bare-bones HA Kubernetes Cluster
6. Installing the CNI/Network Infrastructure
7. Installing the CSIs for Persistent Volumes
8. Installing/Configuring cert-manager
9. Installing/Configuring ArgoCD and GitOps Concepts
10. (this post) Installing/Configuring Authelia/Vault and LDAP/OAuth Integrations
11. Securing Applications with Authelia
12. Keeping your cluster up-to-date

Github for example configuration files: rocket357/on-prem-kubernetes

Overview

I'd written up a good portion on authentication and authorization within kubernetes, namely the kube-apiserver bits, as it pertains to a multi-user cluster, but a specific project we had used in the past at work has been archived by the parent corporation and no suitable alternative has come up yet (indeed, at work we had one of our SREs step up to the plate and write a barebones minimalist replacement for it, though sadly that code is not open source at this time). So today, instead of focusing on authentication and authorization for the kube-apiserver, I want to address authentication and authorization for the applications running within kubernetes.

I've mentioned authelia in this series a few times, and today we're going to be getting it installed and configured. But what is authelia, precisely? At its heart, authelia is:

In other words, authelia is the "glue" that goes between your identity system and your applications. It's a convenient way to add authentication/authorization to applications that might not support your specific identity system...and the best part is, since it supports ingress-nginx, all we have to do to "inject" authelia into the ingress path is to add annotations to the ingress objects we've been creating.

More reasons to use ArgoCD to manage ingress objects

Yesterday I ran into some issues with one of the helm charts I was installing. It seems the chart was written with ingressRoutes in mind (a "vendor specific" alternative to the official kubernetes ingress objects), but did contain support for ingress objects as well. Unfortunately, the ingress template was missing some required functionality, so I provided the team with a PR to add an annotation and a spec item that were missing from the ingress template so it would work. In the interim while waiting for the PR to be approved and merged, I wrote up my own ingress object and deployed it with ArgoCD to the cluster so I could use the application I'd installed. I got to thinking about Authelia and the annotations the ingresses would require, and decided I'd deploy applications within ArgoCD as two components:

1. The main application helm chart deployment
2. a "dependencies" deployment of my own code

The main helm chart deployment in ArgoCD is something I covered in the previous blog post in this series. It's a multi-source deployment, with one source being the public helm chart and the other source being my own private values file for configuring the helm chart. In my values file, I turn off the ingress supplied by the chart. I can also have this set to auto-sync if I wish, though the targetRevision field is going to keep us locked to the specific application version until we update, so at some point we'll need a notification system for chart/application updates...more work for future me!).

In the "dependencies" deployment, I deploy whatever else the application needs (postgres, redis, etc...) as well as the ingress object with the authelia annotations and any customizations necessary (cert-manager, etc...). This is far from ideal, of course, as we're duplicating efforts (writing ingress objects when they *should* exist in the helm charts we're already installing), but ingresses are a bit of a special case since they define the "edge" of the cluster, making traffic possible between the outside world and services within the cluster. That, and there are dozens of ways you can get traffic from outside the cluster to inside the cluster. The opposite side of the spectrum is a bit of a special case from time to time, as well, as some helm charts out in the wild (looking at you, TrueCharts) have dependencies on other operators I'm not using, so I turn off the dependencies for those operators and point the applications to my own operator-controlled resources. (To be fair, TrueCharts are not written with vanilla kubernetes in mind, and as such are pretty much unsupported for our use-case, which is unfortunate as many apps I've wanted to install don't have a public "official" chart, but they *do* have a TrueCharts chart).

Install Authelia Dependencies

I will not lie, Authelia is not an install for the faint of heart. I had a few issues getting it installed on the first cluster, and I suspect I may have issues getting it installed here as well...but with ArgoCD in place it'll at least be more automated than last time.

The dependencies of authelia are:

1. PostgreSQL backing store
2. Redis Caching layer
3. LDAP authentication backend (for my specific use-case)
4. Vault (not strictly required, but highly recommended)

There may be other dependencies, of course, if you choose different options for Authelia. As they say, YMMV.

PostgreSQL is fairly straightforward, as we have some boilerplate YAML manifests already in place for other postgres instances in this cluster. Just copy one over to authelia/, give it an appropriate name, and edit it to put it in the authelia namespace (with appropriate name, username, database name, storagesize, etc...). That can be added to a new ArgoCD "authelia dependencies" application and deployed. Once it's up and running, we can shift our focus to Redis. Ironically, one of the dependencies of ArgoCD (which is running in the cluster currently) is a redis cluster...and it's not operator-managed. Doh.

I toyed around with the official redis operator for a bit, but it seems serious overkill for a simple caching layer for authelia sessions, so I've opted to follow the authelia install guide and stand up a "standalone" redis server instead of an HA cluster. You may see it fit to install the redis enterprise operator, but I don't.

The next big issue I see is that the values file (which we'll be storing in git) needs the passwords for all of the databases and such that it connects to. This is a big no-no in gitops, so first we need to enable the Vault secrets injection bits.

Hashicorp Vault is a server to:

It can be installed via helm as well, so let's setup the vault helm repo in ArgoCD and get to installing it. I had to adjust the caBundle to get vault to accept my ldap configuration, but aside from that the install is fairly straightforward. Once complete, we need to unseal vault, then create a secret engine to hold the details for authelia:

This is going to spit out five "unseal keys" and a root token. These are the literal keys to the kingdom. If they fall into the wrong hands all bets are off, as the decryption of your data will be trivial. On the flip side, if you lose them, well, your vault data is lost as well. Keep them in a safe, secure place.

Now we need to login to vault (with the root token) and setup ldap authentication. This is a fairly lengthy process, so it's tempting to just use the root token for everything. Resist the urge. Just setup the vault ldap integration, login as your primary user, and bob's your uncle. Setup a secrets engine of type "kv" (key-value) and name it authelia (or whatever, your call). Create secrets in here that follow authelia's templates in the default values file. Authelia has templates for the specific passwords you'll need in the default values file.

The next piece in Vault integration is setting up Kubernetes Authentication so pods in your kubernetes cluster can utilize their ServiceAccounts to authenticate to Vault. This way, you can leverage the vault-secret-injector (which runs as either an init container, or a sidecar container, or both (depending on your needs) to your application's main container).

And bring a lunch, because this is a non-trivial piece of machinery that, given Vault's love of path fuckery can easily blow your entire afternoon. Just to get a few passwords safely stored in your pod so you don't have to put plaintext passwords in your git repo (resist the temptation, and use Vault! It'll come in handy in the future as well).

Back to Authelia!

With PostgreSQL, Redis, LDAP, and Vault installed and working, we can get back to installing authelia. Oh, you'll also need a few randomly generated "keys" (long strings preferably over 64 characters) and an smtp account for sending outgoing emails (password reset, multi-factor auth enrollment, etc...). If any of the above is incorrect, authelia will bomb on startup and log a message. Annoying, yes. But would you really want a piece of code that's supposed to keep your applications safely locked behind a login to be acting up? I wouldn't, so once it's up and running let the accomplishment sink in. See if you can authenticate against your LDAP server through authelia, then set up your MFA when prompted. Then laugh when it redirects you to a non-existent domain (because we haven't set that portion up yet).

What's Next?

Next up we're going to step through protecting some of your applications with Authelia (Hello, vault). Then we'll see how many other apps we can break in the process of getting them secured as well.

Cheers!