Architecture Overview

This describes the system as it stands in V4 (the current version; see Version History for the evolution). It’s organized by layer, then by how a single request actually traverses those layers, since the layer diagram alone tends to hide where the actual coupling and failure points are.

Layers

Layer	Tool	Responsibility
Provisioning	OpenTofu (2 root modules)	Declares compute guests, network attachments, and DNS records; owns the “what should exist” statement
Configuration	Ansible	Brings a provisioned host to a known configuration state — packages, users, systemd units
Compute	Proxmox cluster (3 nodes), HA	Runs VMs and LXCs; handles placement and failover
Storage	ZFS-backed NAS, NFS export	Backs shared storage for the cluster; single source of truth for data that needs to survive any one node
Application runtime	Podman Quadlets (systemd units)	Runs containerized services with lifecycle managed by systemd, auto-updating from a registry
Edge	Caddy reverse proxy, CrowdSec	Terminates TLS, routes to the correct service, detects and blocks abusive traffic
Backup	Proxmox Backup Server	Scheduled backup of cluster guests

Each layer is deliberately narrow. Provisioning doesn’t configure anything beyond what’s needed to hand a host to Ansible. Ansible doesn’t manage anything Podman/systemd should own once a host is live. The edge layer doesn’t know anything about storage. Layer boundaries were drawn along “what changes independently of what” — see Provisioning for the reasoning behind the specific split.

Data path

A request for a hosted service follows this path:

client
  │
  ▼
Caddy (reverse proxy, TLS termination, wildcard cert)
  │  — CrowdSec evaluates the request in-line —
  ▼
Podman Quadlet container (systemd-managed)
  │
  ▼
application logic
  │
  ▼
NFS mount → ZFS-backed NAS  (for anything needing persistent state)
  │
  ▼
response traverses back up the same path

TLS terminates once, at Caddy, using a single wildcard certificate rather than a certificate per service. A single credential and renewal path simplifies the common case, at the cost of one failure point for all sites — see Version History for the reasoning. CrowdSec sits in-line at the edge, evaluating requests before they reach any application container, so detection and enforcement happen once for every service rather than being re-implemented per application. Everything behind Caddy is on the internal network; nothing routes around it.

Persistent state, where a service has any, resolves through NFS to the shared ZFS backend rather than to local node storage. This is what makes HA meaningful: a container’s data isn’t tied to the node it happens to be running on, so a failover doesn’t leave the guest running with none of its own state.

Compute: cluster and HA

The Proxmox cluster has three nodes, and HA policy is set per guest with an explicit placement list rather than “run anywhere.” That’s a deliberate constraint: it means a given guest’s failover targets are known and can be reasoned about, at the cost that if every node on that guest’s list is down, the guest stays down — HA reduces that probability, it does not eliminate it.

LXCs are preferred over full VMs (including Docker-in-a-VM) wherever the workload doesn’t specifically require a separate kernel or an OCI runtime — the density advantage has held since V2. Where an OCI image is the natural packaging format, it runs as a Podman Quadlet inside an LXC or VM rather than as a bare podman run invocation, so its lifecycle is managed by systemd like everything else on the host.

Storage: shared, not distributed

Storage is centralized on a single ZFS-backed NAS, exported over NFS, rather than distributed across cluster nodes (no Ceph, no GlusterFS). That’s a deliberate simplicity trade: one well-understood, well-backed-up storage system is easier to reason about and recover than a distributed storage layer with its own quorum and failure modes to manage — at the cost that the NAS itself is a single point of failure for any service that needs persistent state. Backups (Proxmox Backup Server) and planned offsite replication (see Version History) are the mitigations for that concentrated risk, not a claim that the risk doesn’t exist.

Datasets on the NAS are created only when there’s a concrete reason tied to a ZFS feature — a distinct snapshot schedule, quota, replication target, or recordsize. This is covered in more depth in Provisioning.

Edge: one proxy, one certificate

Every externally reachable service sits behind a single Caddy instance, which means one place to configure routing, one certificate to renew, and one point where CrowdSec can evaluate traffic before it reaches any application. The cost of that centralization — a single credential and a single renewal path serving every site at once — is treated as acceptable because it’s monitored and because the alternative (per-service certificates and per-service edge configuration) reintroduces exactly the kind of per-service manual drift this whole rebuild was meant to remove.

What to check if this design is doing its job

• tofu plan against either root module should report no changes on a clean run — a diff means either an out-of-band change happened, or state doesn’t reflect reality.
• A guest’s HA status and placement list should match its actual availability requirement — a guest that’s actually critical but not HA-managed, or one that’s HA-managed with a placement list that doesn’t actually offer redundancy, are both signals the config has drifted from intent.
• Certificate expiry for the wildcard cert is the single highest-leverage thing to monitor at the edge layer, given how much depends on that one renewal succeeding.

See Networking for how traffic actually reaches Caddy, and Provisioning for how every layer above is actually declared and applied.