Provisioning

This is the layer that turned the hand-built, undocumented cluster state (V3) into the declared and reproducible one (V4, current). It covers the toolchain, how it’s structured, and — more importantly than either — how reproducibility is actually verified rather than assumed. See Version History for the evolution.

Toolchain

Tool	Owns
OpenTofu	Guest existence: VMs, LXCs, network attachments, DNS records
Ansible	Host configuration: packages, users, systemd units, file layout
Podman Quadlets	Application lifecycle within a host, as systemd units

The boundary between OpenTofu and Ansible is deliberately the same boundary as “does this require destroying and recreating a resource, or can it be applied in place.” OpenTofu owns anything where the answer is the former; Ansible owns the latter. Blurring that line — letting Ansible manage things OpenTofu should own, or vice versa — is the fastest way back to the V3 problem of nobody being sure which tool is authoritative for a given piece of state.

OpenTofu: two root modules, remote state, locking

Infrastructure is declared across two OpenTofu root modules rather than one. The split follows blast radius and change frequency: one module changes rarely and covers foundational, slow-moving resources; the other changes often and covers guest-level resources that get added, resized, or removed regularly. Keeping them separate means a routine guest change plans and applies against a small, fast state file, and a mistake in the frequently-changed module can’t corrupt state for the foundational one.

State for both lives remotely in Cloudflare R2, with locking enabled. Two properties this buys:

• No local state file. State isn’t tied to whichever machine happened to run the last apply, so any authorized machine can plan and apply against the same source of truth.
• No concurrent-apply corruption. Locking means a second tofu apply started while one is already running fails fast with a lock error, instead of racing the first and leaving state inconsistent with reality.

The cost of the two-module split: there’s no single tofu plan that shows a unified picture across both modules, and a change that depends on both (the common one has to apply before the frequent one can reference its output) has to be sequenced by hand rather than resolved automatically by a single dependency graph.

# verification, not a setup step
tofu plan

A clean tofu plan against unchanged infrastructure should report no changes. Any diff here is a signal — either an out-of-band change happened outside of OpenTofu, or the state doesn’t reflect the real infrastructure — and is treated as something to investigate before the next intentional apply, not something to apply through.

Ansible: host configuration

Once OpenTofu provisions a guest, Ansible brings it to a known configuration state — base packages, users, systemd units, and anything else needed before an application can run on it. Playbooks are written to be safely re-run: running the same playbook against an already-configured host should report changes only where the host’s actual state has drifted from what’s declared, and nothing otherwise.

# verification, not a setup step
ansible-playbook site.yml --check --diff

A --check --diff run that reports pending changes against a host that was configured by the same playbook run yesterday means one of two things: something changed on the host outside of Ansible, or the playbook itself isn’t actually idempotent. Both are worth fixing before they compound.

Podman Quadlets: application lifecycle

Applications run as containers defined via Quadlet unit files rather than started with ad hoc podman run commands. A Quadlet file is a .container (or .volume, .network, etc.) unit that podman-system-generator turns into a normal systemd service — which means the container’s lifecycle is managed with the same tools as everything else on the host: systemctl start/stop/restart, journalctl for logs, and normal systemd dependency ordering against other units.

AutoUpdate=registry is set so containers can update themselves against their source registry on a timer, without a person running a manual pull-and-restart. That’s a deliberate trade of hands-off patching for reduced control over exact rollout timing.

# verification, not a setup step
systemctl status podman-auto-update.timer
podman auto-update --dry-run

The dry run shows which containers have a newer image available without actually updating anything — useful for confirming the auto-update mechanism itself is working before trusting it to run unattended.

Dataset design rule

ZFS datasets on the storage backend are created only when there’s a concrete reason tied to a ZFS feature — a rule that generalized well enough from the current version (V4) that it’s worth stating in full here:

A new dataset is justified when a piece of data needs any of the following, differently from its logical parent:

• a distinct snapshot schedule (different retention or frequency)
• a distinct quota
• a distinct replication target or policy
• a distinct recordsize (large sequential data vs. small random I/O have different optimal values)

“This feels like its own thing” is explicitly not on that list. A dataset created for reasons of feeling rather than one of the four properties above tends to be the first kind of sprawl that shows up when auditing the pool later, with no way to tell in hindsight why it exists as a separate dataset at all.

# verification, not a setup step — for each dataset, confirm it differs
# from its parent on at least one of the four properties above
zfs get -o name,property,value,source recordsize,quota,com.sun:auto-snapshot <dataset>

If a dataset’s relevant properties are all inherited from its parent with no explicit override, that’s a signal it may not have needed to be a separate dataset in the first place.

Applying a change: the shape of a normal workflow

1. tofu plan against the relevant root module — review the diff against intent, not just against “does it look reasonable.”
2. tofu apply once the plan is confirmed to match intent.
3. ansible-playbook site.yml --check --diff against the affected host(s) — confirm the expected changes, and only the expected changes.
4. ansible-playbook site.yml to apply.
5. If the change affects a Quadlet unit, systemctl daemon-reload followed by the relevant systemctl restart for that unit — Quadlet files are regenerated into systemd units on reload, not on save.

None of these steps depend on remembering what was done last time. That’s the property this whole layer exists to guarantee, and the one that earlier versions (V1 through V3) didn’t have. See Version History for why that matters.