Skip to content

Architecture overview

A small homelab that runs like a real one, with GitOps, observability, and a controlled blast radius.

Layout

Internet
  Cloudflare (DNS, proxy for select names)
    │
    ▼
LAN 192.168.1.0/24
  NPM (TLS termination, reverse proxy)
    │
    ▼
  MetalLB pool 192.168.1.225-240
    │
    ▼
  Kubernetes cluster (kubeadm, Calico, ArgoCD)
    apiserver VIP   192.168.1.20:6443
    k8cluster1      192.168.1.90  ── pve
    k8cluster2      192.168.1.89  ── pve2
    k8cluster3      192.168.1.91  ── pve2
                       │  pod->external = SNAT to node IP
                       ▼
  Postgres 16 VM      192.168.1.123  ── pve
  Uptime Kuma VM      192.168.1.129  ── pve  (out-of-cluster monitor)
  TrueNAS             192.168.1.15   (iSCSI: VM disks, NFS: K8s PVCs + media)
  Proxmox pve        192.168.1.10
  Proxmox pve2        192.168.1.11

A higher-fidelity diagram and screenshots live in Visuals.

Hosts and roles

Host IP Role
pve 192.168.1.10 Proxmox node, hosts most VMs
pve2 192.168.1.11 Proxmox node, hosts the rest
K8s API 192.168.1.20 kube-apiserver VIP
k8cluster1 192.168.1.90 K8s worker / control plane
k8cluster2 192.168.1.89 K8s worker / control plane (on pve2, with k8cluster3)
k8cluster3 192.168.1.91 K8s worker / control plane
postgresql 192.168.1.123 Postgres 16, shared backend for Grafana, claude-bridge, and other stateful apps
uptime-kuma 192.168.1.129 Out-of-cluster monitor, live since 2026-09-19 (see ADR 0007). On pve, while the etcd majority sits on pve2. Nothing enforces that, see ADR 0007
truenas 192.168.1.15 TrueNAS. VM disks over iSCSI, Kubernetes PVCs and media over NFS. See Storage
NPM LAN nginx-proxy-manager, TLS termination and routing for *.herro.me

Storage

Undocumented here until 2026-09-19, which is part of why the NFSv4 deadlock was hard to reason about while it was happening.

Everything stateful in the lab lands on one appliance, TrueNAS at 192.168.1.15, over two independent services. That separation is the single most important fact about lab storage, because it is what decides how bad any given NAS problem is.

Service Serves Consumers
iSCSI (ZFS-over-iSCSI) VM disks Every VM in the lab, via Proxmox storage truenas-iscsi
NFS Kubernetes PVCs (/mnt/tank/k8s), media (/mnt/tank/media) k8s-nfs StorageClass, Jellyfin and the *arr stack

They fail independently. On 2026-09-19 NFSv4 wedged completely for eleven hours while iSCSI kept serving, so every VM stayed up and only Kubernetes workloads with NFS PVCs were affected. That is the only reason the recovery was possible: the VMs that did the recovery were themselves running off the broken appliance, over the service that still worked.

The path

  • Storage traffic does not use the LAN. There is a direct 10G NIC-to-NAS link on 10.10.12.0/24 (enp11s0 on pve, enp3s0 on pve2), which is also the Proxmox migration network.
  • iSCSI portal 10.10.12.2, target iqn.2005-10.org.freenas.ctl:proxmox-cluster, backed by tank/proxmox_zfs_vms with a 16k block size.
  • The Kubernetes StorageClass k8s-nfs currently mounts with nfsvers=4.1, hard, timeo=600, retrans=2 (k8s-argocd/infrastructure/nfs/values.yaml). Moving it to NFSv3 is the pending fix from the incident (k8s-argocd#213): v3 is stateless, with no delegations, callbacks or sessions, so that entire deadlock class disappears.

v3 removes the deadlock class, not the ability of clients to wedge

The version change and the hard option are different problems. NFSv3 kills the callback and session machinery, so 2026-09-19 cannot recur in that form. But with hard, a stalled server still parks processes in uninterruptible sleep with no signal able to kill them. That is what a load average of 76 at 4% CPU was. Read "move to v3" as "that deadlock goes away", never as "clients can no longer wedge".

There are two NFS paths, and only one of them has mount options

The Kubernetes PVC path goes through the provisioner above. The media share does not. k8s-argocd/apps/media/media-pv-nfs.yaml mounts 192.168.1.15:/mnt/tank/media with no mountOptions at all, so it negotiates whatever the kernel picks.

Changing infrastructure/nfs/values.yaml alone therefore leaves media on NFSv4.x. Any version migration has to touch both.

The pool

tank is four mirrored vdevs on SSDs, with a separate log (SLOG) and cache (L2ARC) device, roughly 14.7 TB raw.

Not all the drives are trustworthy

Three Inland SSDs from one bad batch repeatedly drop off the SATA bus, and mirror-3 pairs two of them. The drive stops answering, the link cannot be re-established, and it stays gone until a power cycle or a reseat. It is a controller or firmware hang, not media wear: no reallocations, no bad sectors.

The trigger is not known. All three failures are on firmware VE1R9204; the two VE1R9004 siblings in the same chassis on the same controllers have never failed. That correlation is strong. A load correlation is not established, so dmesg on the NAS is part of triage for anything storage-shaped regardless of how busy the pool is.

They are also slow. Measured 2026-09-20 with dd ... iflag=direct: 176 MB/s on a healthy Inland versus 461 MB/s on a WD Blue on the same bus.

The Postgres backups live on the appliance you would need them to recover from

ADR 0002 ships pgBackRest to a TrueNAS dataset. That is the direct corollary of "everything stateful lands on one appliance": the database, its backups and every VM disk share a single failure domain. The periodic restore drill proves the backups are restorable, not that they are reachable when the NAS is the casualty.

Boot order depends on the NAS

pve-guests cannot start VMs until the TrueNAS API answers, which is what wait-truenas-api.sh exists for. A hypervisor that boots faster than the NAS will otherwise try to start VMs whose disks do not exist yet. This bit during the 2026-04-13 outage and it is written up there in detail.

Node-local storage, and one trap

Each Proxmox node also has local LVM-thin storage, which is what VMs are evacuated onto when the NAS needs a reboot:

Node Storages
pve local, SSD1, SSD2, isos
pve2 nvme_1tb, ssd_1, ssd_2

SSD1 and ssd_1 also hold the etcd data disks

They are QLC. Putting bulk sequential writes there drove etcd WAL fsync from 74 ms to 7,627 ms and took the Kubernetes control plane down for about sixteen minutes on 2026-09-19.

Nothing in Proxmox marks these as latency-critical. It shows free space. The constraint exists only as operator knowledge until terraform-quasarlab#24 codifies the etcd disks.

Kubernetes cluster

  • 3 nodes, kubeadm-built, single control-plane endpoint at 192.168.1.20:6443.
  • Calico CNI, pod CIDR 10.244.0.0/16.
  • MetalLB in L2 mode announcing from the pool 192.168.1.225-240 (see ADR 0001).
  • ArgoCD reconciles the cluster from Git. App-of-apps pattern, source repo k8s-argocd.
  • Stakater Reloader watches ConfigMaps and Secrets and rolls Deployments when they change.
  • External Secrets Operator pulls credentials from outside the cluster so secrets never live in Git.

What runs where

Namespace What Notes
argocd ArgoCD HA + Redis HA mode, redis-ha-haproxy
monitoring kube-prometheus-stack, Loki, Vector, Alertmanager, Grafana Discord notification proxy lives here
metallb-system MetalLB controller and speakers L2Advertisement covers the whole pool
external-secrets ESO controllers Backed by a secret store outside the cluster
reloader Stakater Reloader reloader.stakater.com/auto: "true" triggers rollouts
media *arr stack, Jellyfin, qBittorrent, Jellyseerr, NZBGet 192.168.1.226 shared LB IP
automation claude-bridge HITL Discord bridge LB at 192.168.1.235
trading Trading dashboard + API NQ-bias-engine
science Dask cluster, lsdb workloads LSDB (large scale astronomy databases)
dashboard Homepage Lab landing page
security Falco runtime detection Discord webhook for alerts

The full layout is in k8s-argocd.

Why this shape

A few decisions worth calling out, with longer rationale in Decisions:

  • External Postgres VM instead of in-cluster CloudNativePG. Stateful data is the most expensive thing to lose, and I wanted backups, restore drills, and major-version upgrades to be boring even if K8s broke. (ADR 0002)
  • MetalLB L2 instead of BGP, because the home network is a single L2 segment and there is no router I trust to peer with. (ADR 0001)
  • ArgoCD instead of Flux, mostly for the UI when explaining the lab to other humans, and the App-of-Apps pattern fits how I think. (ADR 0003)
  • Discord as the alert sink because it is where I already live. A small custom proxy translates Alertmanager and Falco webhooks into channel-appropriate messages.