Andrew Stuart
- Platform & infrastructure engineer: Kubernetes, Go, cloud, CI/CD
- Running Kubernetes since early 2016 (v1.1/1.2), at home and at work
- Founder of Microcumulus; its product runs on this cluster at ucml.dev
- Tinkerer: home automation, radios, NVRs, far too many disks
Today
- The story: 10 years in 10 minutes
- How it works today
- What broke and what I learned
- Live demo
- Q&A
This etcd dataset has been upgraded in place for 3,787 days. It's older than:
kubeadmHelm 2StatefulSetsCRDsRBACapps/v1cert-managerGateway API
If you replace every plank,
is it the same ship?
- Hardware: three Dell servers replaced the old machines in 2021, and every Node object was re-registered then, five years after the cluster's first API objects.
- Network: Canal → Calico → dual-stack Calico under the Tigera operator
- Ingress: nginx → ingress-nginx → Traefik + Gateway API
- Storage: gluster (for one day) → NFS → Rook → OpenEBS → Rook-Ceph again. The Ceph monitors are now named k, ae, af: that's the 32nd mon.
- Version: about 33 Kubernetes minors, and kubelet compiled from source until 2026
Identity lives in the data, not the machines.
What the cluster looks like today
Live from /api/live: Kubernetes API plus Prometheus, cached 10s. About 275 W at the wall, 295 Mb/s on the wire.
Ten years in ten minutes
1,195 commits by one person. Here's what the git log remembers.
The shape tells the story: build it, let it run, rebuild the foundation, then automate everything so upkeep stops being a chore.
The cluster remembers its own upgrades
I never kept an upgrade log, but I didn't need one. Each release creates built-in objects (APIService registrations, bootstrap RBAC roles, FlowSchemas) the first time it starts, and their creationTimestamp records when it happened.
kubectl get apiservices -o custom-columns=CREATED:.metadata.creationTimestamp,NAME:.metadata.name \
--sort-by=.metadata.creationTimestampHand-written YAML and big ambitions
- First commit Feb 21, 2016: redis, postgres and GitLab as ReplicationControllers
- Mar 2016: "Update to use deployments", back when Deployments were extensions/v1beta1
- Jenkins, Mattermost, gluster volumes, Vault on etcd, dex for SSO, and Prometheus + Grafana (July 2016)
- Workloads pinned to one box by nodeName: htpc, and a Minecraft server with persistence
Security and networking get real
- RBAC in Jan 2017 while it was v1alpha1. That RoleBinding still exists, now served as v1.
- Upgrade to 1.6 and Canal CNI for NetworkPolicy, on the same day
- kube-lego for automatic Let's Encrypt, before cert-manager existed; Helm 2 and Tiller
- Private registry, Nexus, SonarQube, Alertmanager, EFK logging, a Cassandra StatefulSet, Fission functions, an Ethereum node
The quiet years
- Life happened. 68 commits in three years, and only 9 in 2020. The cluster just kept running.
- One weekend in Oct 2018: Canal → Calico, plus MetalLB ("Metal :metal:")
- cert-manager replaces kube-lego; CoreDNS, external-dns, OPA; the first Rook
- May 2020: "ALL THE UPDATES", most likely a jump to 1.18. Commit messages shrink to "Stuff".
New hardware, same cluster
- Three Dell servers join in Sep 2021 and the old machines are drained away. The etcd data moves with them.
- A new CA with cfssl and a GPG-encrypted key; ansible provisions etcd and the control plane
- Helm 3 charts; Rook-Ceph rebuilt (after an OpenEBS detour); Keycloak SSO plus a home-grown oauth2-proxy controller written in Go
- Dual-stack IPv6, Pi-hole, Loki, Jaeger, OpenTelemetry, Velero, Zalando postgres-operator
Supply chain and guardrails
- Harbor becomes the CRI-O pull-through mirror for docker.io and quay.io on every node
- OPA Gatekeeper for admission policy, plus a custom CA-injector webhook so CI runners trust the internal PKI
- Tigera operator takes over Calico, and the cluster moves to 1.23 the same day
- kubelet serving certs from my own PKI, the first try at Argo CD, and ucml.dev for the business
Renovate, and real users
- Renovate (Jan 2023) opens PRs for chart and image bumps, and commit volume jumps almost 5×
- Public, multi-user services: Lemmy (during the Reddit API exodus), Mastodon, Matrix
- Velero + CSI snapshots for backups; Loki in simple-scalable mode; Pyroscope profiling; Argo CD in HA
- git-crypt for secrets in the repo; Istio and Ory experiments; Atuin, Plausible, Postfix relay
The family depends on it now
- Immich for family photos, Frigate NVR on the GPU node (Multus gives it a second NIC for cameras), Home Assistant
- Terraform for DNS zones and the MikroTik router, so infrastructure as code reaches past Kubernetes
- Tried Traefik in May 2024 and went back to nginx five days later
- The busiest day in the repo's history: 70 commits, most of them a Mastodon rebuild
Patch days and a seven-version upgrade
- CVE-2025-1974 "IngressNightmare": patched chart and webhook disabled the day after disclosure
- Rook v1.11 → v1.18 and Ceph Reef → Squid with the data online
- GitLab and Keycloak moved onto postgres-operator; Keycloak HA; Vaultwarden; a Samba NAS on CephFS; a Tailscale subnet router
- Added CLAUDE.md: AI-assisted ops with written runbooks, and read-only checks before any change
Paying down the debt
- Gateway API: about 50 routes moved from ingress-nginx to Traefik, split into public and internal Gateways with LAN-only by default
- kubelet stops being compiled from source; it now installs from release binaries
- 2021 CA renewed, now valid to 2036; PostgreSQL 17; Plausible v3
- A second site on k3s that applies the lessons to a clean, small build, with a DDNS updater written in Go
Every layer has been swapped at least once
The architecture today
Bare metal, done the hard way, on purpose.
Why not kubeadm or k3s?
- kubeadm didn't exist when this cluster was born
- Owning every flag and cert meant understanding every flag and cert
- Upgrades are an ansible run: new release binaries and templated static-pod manifests, one node at a time
- kubectl logs in through Keycloak OIDC; the apiserver trusts auth.stuart.fun
- The trade-off: I am the managed service. The CA expiry in 2026 was mine to catch.
The second site uses k3s. I pick the tool to fit the job.
# ansible/roles/kubehost — per control-plane host
- name: kubehost | ssl | etcd
shell: |
cfssl gencert -ca stuart-2021-ca.crt \
-ca-key <(gpg --decrypt stuart-2021-ca.key.gpg) \
-profile peer -cn "etc{{etcnum}}.astuart.co" \
-hostname kubernetes.default.svc,10.254.0.1,... \
client.json | cfssljson -bare etc{{etcnum}}
- name: kubehost | manifest | apiserver
template:
src: kube-apiserver.yml.j2
dest: /etc/kubernetes/manifests/kube-apiserver.yml
- name: kubehost | systemd | start/restart etcd
systemd: { name: etcd, state: restarted }Real workloads for real people
Family
Immich (photos) · Frigate NVR with GPU detection · Home Assistant · ESPHome · zigbee2mqtt · Vaultwarden · Plex and friends · Samba NAS
Community & friends
WordPress sites for family and friends' businesses · about 15 domains · Plausible analytics · Terraria & Factorio servers · formerly Lemmy, Mastodon and Matrix
Business & dev
Microcumulus (ucml.dev) · two GitLab instances + CI runners · Harbor · Argo CD · 8 Postgres clusters · Garage S3 · a Go oauth2-proxy controller
What broke, and what it taught me
Ten years means everything has failed at least once.
The migration that quietly opened the doors
What happened
ingress-nginx → Traefik Gateway API, about 50 routes. Every route was ported, but nginx's global RFC1918 allowlist lived in a ConfigMap, not on any route. For about an hour, ~30 internal apps (Prometheus, the k8s dashboard, Pi-hole, the *arr stack) were reachable from the internet.
Fix, same day
A lan-only IPAllowList middleware attached to every non-public HTTPRoute, and externalTrafficPolicy: Local so the allowlist sees real client IPs instead of SNAT'd node IPs. I verified it by narrowing the allowlist and checking for a 403 from the LAN. Public became opt-in.
A rotation isn't done until everyone reloads
What happened
The cluster CA expired on 2026-05-16 and was re-issued with the same key. The API server was happy, but every long-running process had cached the old CA cert. For four months, quietly: Prometheus service discovery froze, the Ceph CSI provisioners and two MetalLB speakers lost the API, CRI-O on one node couldn't pull from the internal registry, and Argo CD went dark.
How I found it and fixed it
The deck's own metrics never showed up in Prometheus. Its logs said x509: certificate has expired. A sweep of every pod started before the renewal found 14 more. Rolling restarts, one component at a time, checking the VIP and HEALTH_OK between steps. I deliberately left Argo CD alone: it has auto-sync and prune enabled, and months of drift need a diff review before it reconnects.
The outage nobody noticed (until I did)
What happened
stuart.fun moved to Cloudflare nameservers, but the ACME issuer only knew Route 53. Eight certs quietly expired. The sites stayed up because Cloudflare's edge doesn't verify the origin cert in ssl=full mode.
Fix
Added a Cloudflare DNS-01 solver selected by dnsZones. Selector-matched solvers win, so no Certificate needed editing. Stale TXT records from a cert-manager bug were cleaned up by hand.
Hand-edits are future outages
What happened
Five Gateway listeners had been added by hand in-cluster and never committed. A routine helm apply dropped them. Routes went NoMatchingParent, and external-dns (policy=sync) deleted their DNS records within a minute.
Fix
Moved those hosts to self-contained Ingresses: the TLS block drives the cert and the rules drive DNS. No central list to drift from. Git became the only source of truth again.
The chicken-and-egg power outage
After a power cut, nothing came back. Every node pulled images through Harbor, which ran on the cluster. DNS came from Pi-hole, which also ran on the cluster.
The fix was a runbook, committed to the repo as if-breakers-broke.md:
Upgrade debt compounds
How it got bad
In May 2025 I bumped the Rook CRDs to v1.16 but left the operator on v1.11.9. By October the cluster was seven minors behind, still on Ceph Reef, and carrying tens of TiB of family data with no second cluster to fail over to.
Paying it down
HEALTH_OK first. Then CRDs → common → CSI operator → operator, and Rook rolled mons → mgrs → MDS/RGW → OSDs one at a time while the data stayed online. I pinned timestamped Ceph images, fixed the toolbox last, and wrote the procedure into the repo.
The commit log doesn't lie
IngressNightmare: patched chart plus the published mitigation (webhook off) the morning after disclosure. Know the mitigation, not just the fix.
Deleting a CRD cascades to every custom resource. Now Argo owns its CRDs explicitly, and ownership is pinned.
An allow-list that only knew IPv4 met a dual-stack world. Test both families.
Swapping a CNI under running pods: plan a maintenance window, then plan another.
One of 70 commits on the busiest day, mostly a Mastodon rebuild. Debugging charts by pushing to git is slow, so now I use helm template and diff locally first.
The 2021 CA expired in May 2026 and Vault-issued renewals stalled. It was re-issued with the same key so existing leaf certs kept verifying. Valid to 2036.
What I found while writing this talk
Preparing this talk was an audit. Here's what it turned up, and what's next:
Backups: silently failing
Every Velero run to Backblaze B2 fails on an x-amz-tagging header, probably from an unpinned AWS plugin image. A backup job nobody watches doesn't protect you. Next: pin the plugin and alert on the last successful backup.
GitOps: dark since May
Argo CD's 17 apps have shown Unknown since May: a stale CA and an expired Git token. Next: diff all 17 apps before reconnecting, then alert on sync status.
Fossils
CoreDNS is still 1.3.1 from 2019, cert-manager 1.11 is from 2023, and Bitnami's 2025 catalog change broke four images. Next: let Renovate cover the control-plane add-ons too.
Fixed while writing
11 components stuck on the old CA were restarted (Argo CD deliberately left). A crashed Ceph mgr module was failed over, and Ceph is back to HEALTH_OK. Prometheus discovers pods again.
Principles, paid for in outages
- Upgrade continuously, in place. Small steps, health-gated, automated by Renovate.
- Git is the only source of truth. Controllers will enforce whatever git says, including the gaps.
- Secure by default. LAN-only unless declared public; SSO everywhere; a PKI I understand.
- Observe the internals, not just the front door. Alert on cert expiry, backup age and sync status, not just uptime.
- Know your bootstrap graph. Keep a runbook for the day everything is off.
- Write it down for future-you. Commit messages and runbooks in the repo, readable by people and by AI pair-programmers.
Let's poke it
① Archaeology
The oldest objects still in etcd
② Kill this deck
Delete the pod serving these slides
③ Ship it
A new public HTTPS service in about a minute
Watch the badge in the bottom-right corner →
Questions?
days, one etcd
pods, right now
homelab.stuart.fun
The oldest objects still in etcd
Written as extensions/v1beta1 and rbac/v1alpha1, and still served today as apps/v1 and rbac/v1 after about 33 minor upgrades. The apiserver converts on read.