Networking

Note

Actual IP ranges, VLAN IDs, and hostnames are replaced with placeholders below. The topology pattern and reasoning are accurate; the specific addressing is not, and is left as a fill-in for anyone adapting this pattern to their own network.

Backbone

The cluster and storage nodes sit on a 10G backbone: each Proxmox node and the NAS have 10G SFP+ downlinks into a switch capable of line-rate 10G across all of them simultaneously. Everything else — general household and client traffic — runs at 1G, uplinked to the same switch fabric. The 10G tier exists specifically for the two links that actually need it: cluster-to-cluster traffic (migration, corosync) and cluster-to-storage traffic (NFS).

Segment	Link speed	Why
Proxmox node ↔ Proxmox node	10G SFP+	Live migration and corosync are latency- and throughput-sensitive
Proxmox node ↔ NAS	10G SFP+	NFS is the storage backend for every guest; this link is on the critical path for nearly everything
Everything else	1G copper	No workload here is throughput-bound at 1G

Link aggregation (LACP) is used on the NAS’s uplink specifically, bonding two 10G interfaces — the NAS is the single busiest and most fan-in-heavy node on the backbone, since every cluster node’s storage traffic converges there. Individual Proxmox nodes are not bonded; a single 10G link per node has not been a bottleneck.

VLAN segmentation

Traffic is split by function, not by device type, into a small number of VLANs:

VLAN (placeholder)	Purpose
10.0.10.0/24	Management — Proxmox web UI, IPMI/BMC access, switch/AP management
10.0.20.0/24	Storage — NFS traffic between cluster nodes and the NAS, isolated from everything else
10.0.30.0/24	Services — application/container traffic, what Caddy actually routes to
10.0.40.0/24	General/client — day-to-day household devices, no path to management or storage VLANs

The storage VLAN is isolated specifically so that NFS traffic — which is trusted implicitly by the NFS server and not authenticated per-request in any strong sense — never shares a broadcast domain with general client traffic. The management VLAN is isolated so that anything with control-plane access to the cluster or switching fabric isn’t reachable from the same segment as everyday devices.

Jumbo frames

MTU 9000 is set on the storage VLAN only — both NIC and switch port config, on every interface that carries NFS traffic between the cluster and the NAS. It is not set anywhere else. Jumbo frames reduce per-packet overhead for the large, sequential-ish transfers NFS tends to produce, but only pay off when every hop in the path has it configured consistently — a single 1500-MTU hop in an otherwise-9000 path causes fragmentation or black-holed traffic rather than a graceful fallback, which is exactly why it’s scoped to one VLAN instead of applied network-wide.

# verification, not a setup step — confirm end-to-end jumbo frames on the storage VLAN
ping -M do -s 8972 <nas-storage-vlan-address>

A response with no fragmentation needed confirms the full path — NIC, switch, and the far end — actually supports the 9000 MTU; a failure here with a smaller -s value that still works points at exactly which hop is misconfigured.

Addressing: MAC-pinned plus DHCP reservation

Infrastructure devices — cluster nodes, the NAS, switches, anything Ansible or OpenTofu needs to reach at a predictable address — get a DHCP reservation keyed to MAC address, rather than a static IP configured on the device itself. This keeps address assignment centralized in one place (the DHCP server) rather than scattered across every device’s own network configuration, while still guaranteeing the address never changes. Client and general-purpose devices get ordinary DHCP leases from the pool.

DNS: split horizon

DNS is split-horizon: internal queries for hosted services resolve to the internal (services VLAN) address directly, while external queries for the same names resolve to whatever’s reachable from outside — routed in through Caddy. This means the same hostname works whether it’s resolved from inside the network or from outside it, without hairpinning internal traffic out to the internet and back in through the edge.

Caution

Split-horizon DNS means the two resolution paths can silently disagree if one is updated and the other isn’t — an internal record change that isn’t mirrored externally (or vice versa) produces a service that works from one side of the network and not the other, with no error message pointing at DNS as the cause.

What to check

• dig the same hostname from inside and outside the network and confirm the answers are the ones expected for each side — this is the fastest way to catch split-horizon drift.
• Confirm jumbo frames end-to-end with the ping -M do -s 8972 check above before trusting NFS throughput numbers; a partially-configured jumbo frame path degrades rather than fails outright, which makes it easy to miss.
• Confirm a device’s DHCP reservation actually matches its current MAC before assuming an address is stable — a replaced NIC or a device swap silently breaks the reservation.