Homelab on Joey's Site

GitOps in Practice: What's Actually Automated, and What's Still Manual

Sun, 16 Aug 2026 09:00:00 -0600

Homelab Kubernetes Series: 1. Intro · 2. Installation · 3. Configuration · 4. Networking · 5. Storage · 6. Workloads · 7. ArgoCD · 8. GitOps

Recap

Last post covered getting ArgoCD installed, including a rendering bug that quietly left some Applications running outside GitOps entirely. To close out the series, here’s a snapshot of what GitOps actually looks like day to day in this homelab right now, not the aspirational version, the real one.

What’s genuinely under GitOps today

Auto-sync is on everywhere, with both prune and selfHeal enabled on every Application, no exceptions. In practice that means: edit a manifest in git, push, and within a few minutes the cluster matches. Delete a resource from git, and ArgoCD removes it from the cluster. Edit something in the cluster directly instead of in git, and ArgoCD quietly reverts it back to match git on the next reconciliation loop. That last one took some getting used to. The first time I kubectl edit’d something to test a quick change and watched ArgoCD undo it thirty seconds later, it was a good reminder of what “git is the source of truth” actually means in practice, not just in theory.

Coverage today is honest but incomplete: the workloads reachable from git are the media namespace apps and a handful of others, real, working, auto-syncing. The platform layer isn’t there yet — cert-manager, Traefik, Longhorn, and the observability stack are all still installed and managed the way they were in the installation and configuration posts: by hand, outside git. GitOps started with the easiest, most repetitive layer first and hasn’t gotten down to the foundation yet. That’s next.

Sealed Secrets, running for real

Unlike the ArgoCD image updater below, this one’s live: the Sealed Secrets controller itself is deployed as an ArgoCD Application, sourced straight from its upstream Helm repo rather than my own git repo:

1
2
3
4
5


spec:
 source:
 repoURL: https://bitnami-labs.github.io/sealed-secrets
 chart: sealed-secrets
 targetRevision: 2.18.3

That’s the same controller powering the encrypted-secret workflow from the configuration post, and it’s a good example of ArgoCD managing infrastructure that isn’t my own code at all, just a chart I depend on.

What I tried and walked back: per-project isolation

Early on I wanted ArgoCD’s AppProject concept to segment my workloads: media apps in one project, platform in another, each with its own repo and resource restrictions. I wrote the AppProject, and then wrote the honest verdict directly into the file when I hit the wall:

1
2


# This doesn't work currently. You can only tie 1 repo to 1 project.
# If I had my deployments in separate repos, then I'd create new projects for each.

All of my workload manifests live in one repo, so a project boundary drawn around “one repo” doesn’t actually separate anything. That AppProject was never applied — every Application in the cluster still runs under the default project. Splitting workloads into separate repos per concern is the real fix, and it’s a bigger reorganization than I’ve wanted to take on yet. I’m leaving the dead end in here because “I tried this and the docs made it sound simpler than it turned out to be” is worth writing down, rather than pretending the idea worked the first time.

What’s built but not live: image automation

I added the ArgoCD Image Updater’s Application manifest and annotated one workload (Pi-hole, using its yyyy.mm.x version scheme) to be managed by it:

1
2
3
4
5


annotations:
 argocd-image-updater.argoproj.io/image-list: pihole=pihole/pihole
 argocd-image-updater.argoproj.io/pihole.update-strategy: latest
 argocd-image-updater.argoproj.io/pihole.allow-tags: regexp:^\d{4}\.\d{2}\.\d+$
 argocd-image-updater.argoproj.io/write-back-method: git

write-back-method: git is the part I like most about the design. Instead of silently mutating the live Deployment, the updater would commit the new image tag back to git itself, so even automated version bumps stay visible in commit history. The catch: the Image Updater’s own Application never actually got deployed, so none of this is running yet. The annotations are sitting there, correctly configured, waiting for their controller to exist. It’s on the same list as the app-of-apps fix from the last post.

The gotcha that’s still live: a values.yaml nobody reads

This is the one I want to flag clearest, because unlike the others it’s a silent landmine rather than an inert feature. ArgoCD’s own Helm install has a values.yaml sitting at my repo root, with two settings I actually care about:

1
2
3
4
5


configs:
 params:
 server.insecure: "true"
 cm:
 accounts.admin: "apiKey,login"

But the Application that installs ArgoCD points at charts/argo-cd, and that directory has no values.yaml of its own. My root-level file is never read by ArgoCD at all. It’s a leftover from the very first helm install -f values.yaml I ran by hand, before ArgoCD was managing itself.

Right now those two settings are still active in the cluster, purely because ArgoCD’s reconciliation merges into the existing ConfigMaps rather than replacing them wholesale. The values survive as leftover keys Helm’s chart defaults don’t know to remove. But that’s fragile: if either ConfigMap ever gets deleted and recreated from scratch, both settings silently revert to upstream defaults. server.insecure reverting to false would be the visible one — my Traefik route talks plain HTTP to ArgoCD’s server on the assumption that TLS is terminated upstream, so a default-secure ArgoCD would break the UI behind its own working certificate. The fix is simple: move the file into charts/argo-cd/values.yaml where the Application actually looks for it. That’s genuinely the next thing I’m doing after publishing this post.

Where this goes next

Looking at everything laid out across these gotchas, the real GitOps roadmap for this homelab is:

Fix the app-of-apps rendering bug so every Application actually lives under templates/.
Move ArgoCD’s values.yaml to where its own Application will actually read it.
Split workloads into separate repos (or at least separate paths with real project boundaries) so AppProjects can do something useful.
Deploy the Image Updater for real, now that the annotations are already sitting there configured.
Bring the platform layer — cert-manager, Traefik, Longhorn, observability — into git the same way the workloads already are.

None of that is a rewrite. It’s closing gaps I only found by actually running this for months and watching where reality quietly diverged from what the repo said it should be. That, in hindsight, was the actual point of this series: not a perfect homelab, but what building one for real, mistakes included, actually looks like.

Thanks for following along.

◀ Previous: 7. ArgoCD

Moving to ArgoCD: An App-of-Apps, and the Bug That Taught Me How Helm Rendering Works

Thu, 13 Aug 2026 09:00:00 -0600

Homelab Kubernetes Series: 1. Intro · 2. Installation · 3. Configuration · 4. Networking · 5. Storage · 6. Workloads · 7. ArgoCD · 8. GitOps

Recap

Everything up through the last post got onto the cluster the same way: kubectl apply -f, from whichever terminal I happened to have open, applied in whatever order I remembered to run things. That works right up until it doesn’t. This post is about why I moved to ArgoCD, and a bug in my own setup that I didn’t catch for months.

Why kubectl apply stopped being enough

A few things pushed me here. There was no record of what was actually deployed versus what I’d only ever run once from history. There was no reconciliation — if something drifted from the manifest, or someone (me, running a one-off kubectl edit) changed it directly, nothing would ever notice or fix it. And there was no single place to look and answer “is the cluster in the state I think it’s in.” GitOps fixes all three: git is the source of truth, and a controller in the cluster keeps reality in sync with it.

I followed Micah Bird’s homelab ArgoCD guide as the starting pattern, in a separate repo from the workload manifests themselves.

Installing ArgoCD, and having it manage itself

ArgoCD is installed as a small Helm wrapper chart around the upstream argo-helm chart:

1
2
3
4
5
6
7


apiVersion: v2
name: argo-cd
version: 3.2.1
dependencies:
 - name: argo-cd
 version: 9.1.7
 repository: https://argoproj.github.io/argo-helm

The interesting part isn’t the install itself, it’s what happens right after: ArgoCD manages its own upgrade going forward, as just another Application pointed at that same chart path in the same repo. Once it’s up, I don’t run helm upgrade by hand anymore. I bump the version in git and let ArgoCD reconcile itself.

Ingress needed one non-obvious detail. ArgoCD’s UI and CLI both talk gRPC, which doesn’t play nicely with a plain HTTP-routed IngressRoute, so the route needs a second, higher-priority rule specifically for gRPC traffic, upgraded to h2c:

1
2
3
4
5
6
7


- kind: Rule
 match: Host(`argocd.joeyaxtell.com`) && Header(`Content-Type`, `application/grpc`)
 priority: 11
 services:
 - name: argo-cd-argocd-server
 port: 80
 scheme: h2c

Without that, the web UI loads fine but the CLI and any gRPC-based calls fail in confusing ways.

App-of-apps

The pattern I landed on is the classic app-of-apps: one root Application watches a directory in git, and every file in that directory becomes another Application that ArgoCD manages:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


spec:
 source:
 repoURL: https://github.com/path-to-my-argo-cd-repo.git
 path: apps/
 targetRevision: HEAD
 syncPolicy:
 automated:
 prune: true
 allowEmpty: true
 selfHeal: true

Add a workload, add one small Application file describing where its manifests live and where it should be deployed, and ArgoCD picks it up automatically. Each one follows the same shape: point at a path in my homelab-kubernetes repo, sync automatically, prune anything removed, and self-heal anything that drifts.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


spec:
 source:
 repoURL: https://github.com/path-to-my-k8-manifest-repo.git
 targetRevision: HEAD
 path: media/<app>/
 directory:
 recurse: true
 syncPolicy:
 automated:
 prune: true
 selfHeal: true
 ignoreDifferences:
 - group: apps
 kind: Deployment
 jsonPointers: [/spec/replicas]

That last bit, ignoreDifferences on replica count, exists so that if I manually scale something down for maintenance, ArgoCD’s self-heal doesn’t immediately scale it back up and fight me.

The bug: not every Application file is actually an Application

Here’s the one I didn’t catch for a while, and I think it’s a genuinely useful lesson about how Helm rendering works. My apps/ directory is itself a Helm chart — it has a Chart.yaml — which means ArgoCD renders it as a Helm chart. Helm has a rule I hadn’t fully internalized: only files under templates/ get rendered as manifests. Anything sitting at the chart root is just a file Helm can reference via .Files, but it never gets emitted.

I had new workload Applications sitting at the chart root instead of inside templates/. They were valid YAML, they were committed to git, and the root app-of-apps Application showed as Synced at the exact commit that added them. Every signal you’d normally trust said everything was fine. But because they weren’t inside templates/, Helm silently never rendered them, and they never became real ArgoCD Applications. The workloads they described only exist in my cluster today because I kubectl apply’d them by hand at some point. GitOps for those apps is, right now, an illusion. If I’d edited that file in git expecting a sync to follow, nothing would have happened, because ArgoCD never rendered it as anything to sync.

The fix is a one-line move — apps/pihole.yml becomes apps/templates/pihole.yml — and it’s on my list. I’m leaving the mistake in this post instead of quietly writing around it, because “the sync status looked healthy and I was still wrong” is exactly the kind of thing worth knowing to check for.

Secrets and private repos

ArgoCD needs credentials to pull from a private GitHub repo, which it gets from an ordinary Secret carrying one special label:

1
2
3
4
5
6
7
8


metadata:
 labels:
 argocd.argoproj.io/secret-type: repository
stringData:
 type: git
 url: https://github.com/[GITHUB_REPO_HERE]
 username: [GITHUB USERNAME HERE]
 password: [GITHUB PAT TOKEN HERE]

That label is what tells ArgoCD to treat this Secret as repository credentials rather than just an opaque Secret sitting in the namespace. The real values are created out-of-band, the same way the Cloudflare token was in the configuration post — the committed file is a template, never a credential.

Up next

Getting ArgoCD installed is one thing; actually leveraging it day to day is another. The next post covers what’s really running through it today, what I tried and had to walk back, and where the gaps still are.

◀ Previous: 6. Workloads | Next ▶: 8. GitOps

Running Workloads: From Compose Files to Kubernetes Manifests

Mon, 10 Aug 2026 09:00:00 -0600

Homelab Kubernetes Series: 1. Intro · 2. Installation · 3. Configuration · 4. Networking · 5. Storage · 6. Workloads · 7. ArgoCD · 8. GitOps

Recap

Certificates, secrets, networking, and storage are all in place after the last four posts. This one is about the actual workloads: turning what used to be Docker Compose stacks into real Kubernetes manifests, and the conventions that came out of doing that a dozen times over.

The structural pattern

Nothing fancy here: one folder per application, and inside it a single manifest file with a Deployment, Service, and Ingress concatenated together with ---. No Helm charts for my own apps, no Kustomize overlays, just plain YAML applied with kubectl apply -f. It’s not the most sophisticated pattern, but it maps almost one-to-one onto how I used to think about a Compose service, which made the conversion a lot less intimidating than I expected going in.

A few conventions repeat across nearly every workload I run, mostly inherited straight from the linuxserver.io image conventions I was already used to from Compose:

1
2
3
4
5
6
7


env:
 - name: PUID
 value: "3000"
 - name: PGID
 value: "3000"
 - name: TZ
 value: "America/Chicago"

Consistent UID/GID across every app made the NFS permissions story from the storage post a lot simpler: one user, one group, every app writing to shared volumes without stepping on each other’s file ownership.

What’s actually running

I keep a handful of apps in a dedicated media namespace that I’m not going to detail here. What they do and how they’re wired up isn’t the interesting part of this series. What is relevant to a Kubernetes post: those workloads split across both storage tiers exactly like the last post described (small per-app config volumes on Longhorn, a large shared library on NFS), and one of them needs privileged networking capabilities beyond what the rest of the cluster requires — a reminder that Kubernetes can still handle non-standard networking when you actually need it.

Outside of that namespace, a few things are worth naming specifically:

Pi-hole — DNS for the network, covered in the networking post.
Home Assistant — my home automation platform, with its own Ingress and a persistent NFS-backed config volume.
Heimdall — a simple landing page/dashboard linking out to everything else I run.
Diun — the update-tracking watcher from the configuration post.

What didn’t survive

Not every experiment stuck around, and that’s not really a bad thing. A Kubernetes Dashboard deployment went through three rounds of tweaking over about a week and then got deleted — Traefik plus Grafana plus kubectl covered what I actually needed, and running a second web UI just to look at the cluster stopped earning its keep. A chat-gateway service got built out over several commits — init container, security context, Sealed Secrets for its API tokens, the works — and was pulled a day later once I decided it wasn’t worth the maintenance surface for how often I’d actually use it. Its PVC is still sitting there, unclaimed, which is its own small lesson about cleaning up after yourself.

Neither of those is really a failure story. They’re closer to “built it, used it for a week, decided it wasn’t worth carrying forward,” which is a normal part of running a homelab. I’d rather show that than pretend every manifest I’ve ever written is still in service.

The maturity marker

The clearest sign that this stack has moved from “getting it working” to “operating it” is a single commit that touched twelve deployments at once, adding the failure-tolerance settings from the configuration post — shorter node-failure tolerations, paired with strategy: Recreate everywhere a ReadWriteOnce volume is involved. You don’t make a change like that on day one. You make it after a node actually goes down and you sit there watching five minutes tick by before anything comes back.

Up next

Everything so far has been applied by hand — kubectl apply -f, one file at a time, from whichever terminal I happened to be in. The next post is about why that stopped being good enough, and moving to ArgoCD.

◀ Previous: 5. Storage | Next ▶: 7. ArgoCD

Storage in the Homelab: NFS for Bulk, Longhorn for Anything With a Database

Thu, 06 Aug 2026 09:00:00 -0600

Homelab Kubernetes Series: 1. Intro · 2. Installation · 3. Configuration · 4. Networking · 5. Storage · 6. Workloads · 7. ArgoCD · 8. GitOps

Recap

Traffic can reach a pod as of the last post. This one is about what happens after that: where the data actually lives, and why I ended up running two completely different storage systems instead of one.

The split: NFS vs. Longhorn

I run storage on two tiers, and the dividing line comes down to one question: does this workload need a single-writer block device, or can it live happily on shared network storage?

Tier one is NFS, backed by a TrueNAS box on my network. I provisioned a handful of statically-defined PersistentVolumes against it: one for general app config, one for a large shared media library, and one for another large content volume, all ReadWriteMany and all set to Retain so deleting the claim never deletes the data.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


apiVersion: v1
kind: PersistentVolume
metadata:
 name: truenas-config-pv
 labels:
 type: config
spec:
 capacity:
 storage: 1Ti
 accessModes:
 - ReadWriteMany
 persistentVolumeReclaimPolicy: Retain
 nfs:
 server: 192.168.0.126
 path: /mnt/media/configs
 storageClassName: config

There’s no dynamic NFS provisioner in the mix, so binding a PVC to the right PV is done the old-fashioned way, with label selectors:

1
2
3
4


spec:
 selector:
 matchLabels:
 type: config

A handful of small apps — a dashboard, a home automation platform, an update-tracking service, a DNS server — all share that single config PV, each carving out its own directory with subPath. It’s a simple pattern, and it works well for anything that just wants a folder to read and write config files from, shared across as many pods as need it.

Tier two is Longhorn, MicroK8s’s built-in distributed block storage addon, used for ReadWriteOnce volumes where something wants exclusive, low-latency access to its own disk. Every app in my media namespace that needs a config volume gets its own small dedicated Longhorn PVC (a few gigabytes each) rather than sharing the NFS config volume the smaller apps use.

Longhorn itself lives entirely in the addon layer. microk8s enable turned it on, so the only thing I actually own in git for it is its Ingress, giving me a web UI at longhorn.joeyaxtell.com. That also means I don’t have replica counts, backup targets, or a documented disaster-recovery story committed anywhere in the repo yet. It’s a gap, and it’s on the list.

The migration that taught me the dividing line

I didn’t start out with a clean rule for which tier a workload belonged on. I found the rule the hard way. A handful of containers in my media namespace keep their own local state in an embedded database, SQLite in every case that bit me. SQLite and NFS do not get along: file locking over a network filesystem is unreliable enough that those apps would intermittently stall or throw database-locked errors under normal use.

The fix was a straight storage migration: move each affected app’s config volume off the shared NFS PVC and onto its own dedicated Longhorn PVC instead. The difference was immediate. No more lock contention, because Longhorn gives each pod exclusive block-level access to its own volume instead of arbitrating access across the network. That’s the same reasoning behind putting every media app’s config volume on Longhorn in the first place: anything with a real database wants a real disk, not a network share, even if the network share is easier to set up.

The rule of thumb I use now: bulk, shared, or read-heavy data → NFS. Anything with an embedded database or that’s picky about file locking → Longhorn.

A debug pod for poking at block storage

One annoyance with ReadWriteOnce Longhorn volumes: you can’t just browse them from outside the pod that’s using them the way you can mount an NFS share from anywhere. So I keep a small utility pod around for exactly that:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


apiVersion: v1
kind: Pod
metadata:
 name: pvc-toolbox
spec:
 restartPolicy: Never
 containers:
 - name: toolbox
 image: ubuntu:22.04
 command: ["sleep", "infinity"]
 volumeMounts:
 - name: data
 mountPath: /data
 volumes:
 - name: data
 persistentVolumeClaim:
 claimName: media-config-pvc # change this to the PVC you want to access

The workflow is three commands:

1
2
3


kubectl exec -it pvc-toolbox -- bash
kubectl cp /local/path/to/file pvc-toolbox:/data/path/in/pvc
kubectl delete pod pvc-toolbox

Point it at whatever PVC needs inspecting, exec in or kubectl cp files in and out, then throw the pod away. It’s not fancy, but it’s saved me from writing one-off debugging tools more times than I expected.

Up next

With traffic routed and storage sorted, the next post covers what’s actually running on top of all this — how I converted Compose stacks into Kubernetes manifests, the conventions I standardized on, and a couple of experiments that didn’t survive contact with reality.

◀ Previous: 4. Networking | Next ▶: 6. Workloads

Kubernetes Networking in the Homelab: Traefik, MetalLB, and Pi-hole

Mon, 03 Aug 2026 09:00:00 -0600

Homelab Kubernetes Series: 1. Intro · 2. Installation · 3. Configuration · 4. Networking · 5. Storage · 6. Workloads · 7. ArgoCD · 8. GitOps

Recap

By the last post I had certificates and secrets sorted out. None of that matters if traffic can’t actually reach a pod, so this post covers the path a request takes from a browser on my LAN to a container running in the cluster.

MetalLB: giving the cluster real LAN addresses

Bare-metal Kubernetes has no cloud provider to hand out LoadBalancer IPs for you, which is the gap MetalLB fills. It’s a MicroK8s addon here, so there’s no manifest of my own to show, but the effect is that type: LoadBalancer Services get a real, dedicated IP out of a pool I control instead of sitting in <pending> forever.

Every application in the cluster shares one pattern: it sits behind Traefik, and Traefik gets exactly one of those addresses. The one exception is DNS. Pi-hole needs to be the DNS resolver for the network, on a fixed, memorable IP, answering on the actual DNS port, and routing that through an HTTP reverse proxy doesn’t make sense. So it gets its own dedicated LoadBalancer IP straight from the pool:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


apiVersion: v1
kind: Service
metadata:
 name: pihole-dns
spec:
 type: LoadBalancer
 loadBalancerIP: 192.168.0.241
 ports:
 - name: dns-udp
 port: 53
 protocol: UDP
 - name: dns-tcp
 port: 53
 protocol: TCP

Every device on my network points its DNS at that one address.

Traefik: one ingress, host-based routing

Traefik is installed by hand via Helm rather than a MicroK8s addon:

1

helm install traefik traefik/traefik -f traefik_values.yaml --wait

The values file is intentionally small — a dashboard IngressRoute and one forward-looking setting: the Kubernetes Gateway API provider is turned on.

1
2
3
4
5
6
7
8


providers:
 kubernetesGateway:
 enabled: true
gateway:
 listeners:
 web:
 namespacePolicy:
 from: All

That Gateway API setting isn’t actually doing anything yet. Every workload in the cluster still routes through a regular Kubernetes Ingress, which is the older, more common way to expose a Service. Gateway API is the newer replacement for Ingress, and I turned Traefik’s support for it on so it’s ready if I ever want to move a workload over. For now it’s just sitting there switched on and unused.

The real pattern is one dedicated subdomain per app: app1.joeyaxtell.com, pihole.joeyaxtell.com, grafana.joeyaxtell.com, argocd.joeyaxtell.com, and so on, each with an Ingress that references the same TLS ClusterIssuer from the last post. Traefik terminates TLS, matches the Host() rule, and forwards to the matching in-cluster Service.

DNS: Pi-hole as the front door for hostnames

With Pi-hole holding a fixed LAN address and handling DNS for the whole network, every one of those *.joeyaxtell.com hostnames just needs a local DNS entry pointing back at Traefik’s address. That’s what makes “internal-only, real TLS cert, friendly hostname” work together: Pi-hole resolves the name, Traefik terminates the certificate and routes by host header, and from a browser’s perspective it looks exactly like a normal public site.

Pi-hole is also the clearest example of the Sealed Secrets pattern from the last post in practice — its upstream resolvers and web admin password are stored as an encrypted SealedSecret and pulled in with envFrom.secretRef, instead of sitting in the deployment manifest as plain text.

What’s deliberately not here

No service mesh, no Tailscale or WireGuard overlay. Everything described here is LAN-only, reachable because Pi-hole resolves the hostname and MetalLB/Traefik own the routing. A few workloads in my media namespace have their own networking requirements beyond this, but I’m leaving those out since they’re app-specific rather than cluster networking concerns.

Up next

Traffic can reach a pod now. Next up: what happens to the data once it gets there — the two storage tiers I ended up running side by side, and why.

◀ Previous: 3. Configuration | Next ▶: 5. Storage

Configuring the Cluster: TLS, Secrets, and Learning to Not Lose a Node

Fri, 31 Jul 2026 09:00:00 -0600

Homelab Kubernetes Series: 1. Intro · 2. Installation · 3. Configuration · 4. Networking · 5. Storage · 6. Workloads · 7. ArgoCD · 8. GitOps

Recap

Last time I had three MicroK8s nodes joined into a cluster. That’s a cluster you can kubectl get nodes against, but it’s not yet a cluster you’d trust with anything real. This post covers the configuration work that closed that gap: certificates, secrets, and a couple of lessons I learned the hard way about how the cluster behaves when a node actually dies.

TLS: real certificates for an internal-only cluster

Every host I run is a subdomain of joeyaxtell.com, but none of them are reachable from the public internet — they all resolve to addresses inside my LAN. That rules out the usual HTTP-01 ACME challenge, which needs Let’s Encrypt to reach your server directly. The fix is a DNS-01 challenge through Cloudflare, which proves domain ownership by writing a TXT record instead of serving an HTTP response:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
 name: letsencrypt-cloudflare
spec:
 acme:
 server: https://acme-v02.api.letsencrypt.org/directory
 email: YOUR_EMAIL@EXAMPLE.COM
 solvers:
 - dns01:
 cloudflare:
 apiTokenSecretRef:
 name: cloudflare-api-token
 key: api-token

That’s the production ACME endpoint, not staging. I went straight for real certificates on the first ClusterIssuer I ever wrote, which was a little bold in hindsight — Let’s Encrypt’s production rate limits aren’t generous if you get the config wrong and end up retrying in a loop. It worked out, but staging first is the safer habit.

I proved it worked the same way I proved everything else in this series: a disposable nginx pod at test.joeyaxtell.com, watched until a real certificate showed up in its Secret, then deleted.

Secrets: three stages, and only the first one is embarrassing

My secrets story is basically a timeline of learning why each previous approach doesn’t scale.

Stage 1 — plaintext templates committed to git. The Cloudflare API token secret started life as a literal kind: Secret manifest with api-token: YOUR_CLOUDFLARE_API_TOKEN committed to the repo, meant to be hand-edited locally before applying. Never a real credential in git, but also not a pattern I’d want to repeat as the number of secrets grew.

Stage 2 — imperative, out-of-band kubectl create secret. For a while, secrets simply didn’t exist in the repo at all, just a comment documenting the kubectl create secret generic ... --from-literal=... command I’d run once by hand and never again. It works, but it means the repo doesn’t actually describe the cluster. There’s a whole category of state that only exists in my shell history.

Stage 3 — Sealed Secrets. This is where I landed, and where new secrets go today. The Bitnami Sealed Secrets controller lets you encrypt a Secret client-side with its public key, commit the encrypted SealedSecret to git, and only the controller running in-cluster can decrypt it back into a real Secret. The workflow, including the PowerShell-specific piping since I do this from Windows:

1
2


kubeseal --fetch-cert --controller-namespace default --controller-name sealed-secrets-controller > cert.pem
Get-Content secret.yaml | kubeseal --cert cert.pem -o yaml > sealedsecret.yaml

That finally makes both a secret’s existence and its encrypted value visible in git history, without the value ever being recoverable by anyone who doesn’t hold the cluster’s private key. It’s not applied everywhere yet — that migration is still going workload by workload — but it’s the pattern I reach for now.

Update tracking: an opt-in watcher

Rather than a blanket “check everything for updates” policy, I run Diun with Kubernetes provider support enabled, watching every six hours. Workloads opt in individually by adding one annotation to their pod template:

1
2


annotations:
 diun.enable: "true"

That opt-in model matters. I want to know when something has a new image available, but I don’t want that noise on things I’ve deliberately pinned. When I locked one workload to a specific version and set imagePullPolicy: IfNotPresent so it would stop drifting, the next thing I did was pull the diun.enable annotation back off it. No point getting paged about updates I’ve already decided not to take.

Configuring for node failure

The most useful configuration change I made didn’t happen until months in, after I actually lost a node and watched what happened: nothing, for a long time. Kubernetes’ default tolerance for an unreachable node is generous — pods on a node that goes NotReady or Unreachable aren’t rescheduled for five minutes by default. On a three-node homelab cluster, five minutes of a chunk of your services being down because one box hiccuped is a bad trade.

The fix, applied across every deployment in one pass once I understood the knob:

1
2
3
4
5
6
7
8
9


tolerations:
 - key: "node.kubernetes.io/not-ready"
 operator: "Exists"
 effect: "NoExecute"
 tolerationSeconds: 30
 - key: "node.kubernetes.io/unreachable"
 operator: "Exists"
 effect: "NoExecute"
 tolerationSeconds: 30

That cuts the eviction wait from five minutes to thirty seconds. I paired it with strategy: Recreate on the same deployments, since my persistent volumes are ReadWriteOnce and a rolling update trying to attach the same volume to a second pod before the first releases it just hangs. Recreate tears the old pod down before standing the new one up. It’s slower for a routine deploy, but it’s the only strategy that actually works with single-writer storage.

Up next

With TLS, secrets, and failure handling in place, the next post covers how traffic actually gets from my LAN to a pod — Traefik, MetalLB, and DNS.

◀ Previous: 2. Installation | Next ▶: 4. Networking

Installing Kubernetes: Picking MicroK8s for the Homelab

Tue, 28 Jul 2026 09:00:00 -0600

Homelab Kubernetes Series: 1. Intro · 2. Installation · 3. Configuration · 4. Networking · 5. Storage · 6. Workloads · 7. ArgoCD · 8. GitOps

Recap

In part one I explained why I was moving off a single Docker Compose box and onto Kubernetes. This post covers the actual install: what I picked, what I built it on, and the small role Ansible played.

Picking a distribution

I didn’t want to hand-roll kubeadm for my first real cluster, and I wasn’t ready to fight with Talos’s immutable, API-only model on day one either. I landed on MicroK8s. It’s a single-snap install, it ships HA out of the box once you have three or more nodes, and the addon ecosystem (microk8s enable <thing>) let me turn on pieces of the platform — DNS, storage, observability — without writing them myself while I was still learning what each one did.

There’s a real trade-off there. Addons are convenient, but they’re also opaque. A chunk of what’s running in my cluster today, Longhorn and the observability stack included, was never kubectl apply’d from a manifest I own — it just appeared because I typed microk8s enable observability one night. That gap shows up again in the storage and GitOps posts: anything installed by an addon doesn’t have a home in git, so my repo only holds the edges of those systems (an Ingress here, a ClusterIssuer there), not the whole install.

The hardware

Three nodes, no dedicated control-plane/worker split:

1
2
3
4


[k8s_nodes]
k8-node1.thos.local
k8-node2.thos.local
k8-node3.thos.local

MicroK8s’s HA mode makes all three control-plane and worker at once, so as long as two of three are up, the cluster keeps quorum and keeps scheduling. That was the whole point: no more “the one server’s down and everything’s down.”

Ansible showed up late, and only for two things

If you’re picturing an Ansible playbook that bootstraps MicroK8s from bare metal, that’s not what happened. The cluster was already running real workloads before Ansible ever entered the picture. When it finally showed up, it did exactly two things.

k8_setup.yml installs host prerequisites, and it’s shorter than you’d expect:

1
2
3
4
5
6
7
8
9


- hosts: k8s_nodes
 become: yes
 tasks:
 - name: Install open-iscsi and nfs-common
 apt:
 name:
 - open-iscsi
 - nfs-common
 state: present

Those two packages exist for one reason: they’re what Longhorn and NFS-backed volumes need on the host to attach storage. This playbook is a storage prerequisite, not a cluster bootstrap.

updates.yml is even simpler — apt update && apt upgrade across all three nodes, added about a month later once I got tired of patching each box by hand over SSH.

Everything else — MicroK8s itself, the addons, the Helm installs — was done by hand, one node and one terminal at a time. My “automation” story started small and grew later; it didn’t arrive fully formed.

Prove it before you build on it

A pattern shows up over and over in my early commits: stand up a piece of infrastructure, then immediately deploy something disposable to prove it works before trusting real workloads to it. The first thing I put behind Traefik wasn’t a real app — it was three hashicorp/http-echo containers, just to confirm host-based routing worked. Cert-manager got the same treatment: an nginx pod at test.joeyaxtell.com existed for one reason, to watch a real certificate get issued before I pointed anything real at the ClusterIssuer.

It’s a small habit, but it saved me a lot of debugging-two-things-at-once later. More on both of those in the configuration and networking posts.

Up next

With three nodes joined and quorum established, the next step was making the cluster trustworthy enough to run things on — TLS, secrets, and the conventions I settled on for every workload going forward.

◀ Previous: 1. Intro | Next ▶: 3. Configuration