Version History
Four iterations, each triggered by a structural problem the previous version hit.
Comparison
Section titled “Comparison”| V1 | V2 | V3 | V4 | |
|---|---|---|---|---|
| Compute | 1 shared desktop | 1–2 hand-built hosts | 3-node cluster | 3-node cluster |
| Virtualization | None (Docker) | Proxmox, hand-built LXCs | Proxmox cluster + HA | Proxmox cluster + HA |
| Storage | Local disk | Local ZFS per-node | Shared NAS over NFS | Shared NAS over NFS |
| Provisioning | None | None | None | OpenTofu (2 modules) |
| Configuration | None | None | None | Ansible |
| App runtime | Docker Compose | LXC manual install | LXC manual install | Podman Quadlets |
| Edge | None | None | Manual per-service | Single Caddy + CrowdSec |
| Backup | None | None | Manual | Proxmox Backup Server |
| Services | ~8 | ~20 | ~35 | ~40 |
| Broke on | Single disk failure | No HA, stranded nodes | Config drift on 3 nodes | Cross-module apply ordering |
Timeline
Section titled “Timeline”V1 — Gaming Desktop (single box, Docker Compose)
- One shared desktop running services and personal use concurrently.
- Broke: disk failure took everything. Learned: services need isolation.
V2 — Half Rack (Proxmox + hand-built LXCs)
- Dedicated hardware, Proxmox cluster attempted. Hand-built LXCs via install scripts.
- Broke: no way to move workloads between nodes when one failed. Configuration existed only in the GUI. Learned: cluster needs HA, config needs to be code.
V3 — Full Rack (3-node cluster, shared NFS)
- Three-node Proxmox cluster with HA enabled. Shared ZFS-backed NAS. Still configured by hand.
- Broke: tripled the manual surface. Nobody could say what was actually configured on a given node without comparing notes by eye. Learned: reproducibility must be enforced, not hoped for.
V4 — Infrastructure as Code (current, OpenTofu + Ansible + Podman Quadlets)
- Same physical footprint as V3. Everything above the hypervisor is now declared: OpenTofu provisions guests, Ansible configures them, Podman Quadlets run applications as systemd units.
- 10G backbone for cluster-to-cluster and cluster-to-storage traffic. Single reverse proxy (Caddy) with one wildcard cert and CrowdSec at the edge. Remote state with locking so concurrent applies can’t corrupt state.
- Status: current. See Provisioning for how it’s structured, Architecture Overview for the layers, and Roadmap for what’s deliberately not built yet.
Roadmap
Section titled “Roadmap”Offsite replication
- Snapshots are currently local. Same pool as the working data, so the pool is still a single point of failure. Offsite replication to a separate location removes that, but requires coordination with the backup job schedule to avoid contention. Not built: waiting for a second storage location to exist first.
NVMe redundancy at the boot/cache tier
- Current NVMe is consumer-grade, single device. Higher-reliability alternatives exist (enterprise SSD, NVMe with enterprise controller, or RAID-1 NVMe pairs) but the workload hasn’t justified the cost yet. Revisit after measuring actual failure rate.
Storage monitoring that catches stalls before users do
- Backups and snapshots are working, but the 2026-08-30 NVMe incident (untrimmed SSD, garbage collection stall, NFS hung while SMART stayed green) showed that clean metrics don’t mean the service is actually healthy. Add reconcilers — timers that verify NFS is actually responsive, not just that the pool reports healthy.
Remaining hand-managed network rules as code
- VLAN assignments, switch port config, and DHCP reservations are still documented outside of OpenTofu. These should be in code, not a wiki page.
Rootless Podman as a hardening step
- Containers currently run as root inside the unprivileged LXC. Rootless Podman would add another boundary, but requires coordination with bind-mounted volumes and device passthrough.
CI-driven applies
- OpenTofu plan runs locally. A CI pipeline that runs the plan on every merge, shows the diff, and gates the apply would catch drift earlier.