boma/docs/superpowers/specs/2026-06-18-local-vm-integration-testing-design.md

Local VM integration testing on ubongo (design)

Status: Designed, not built. Resolves docs/TODO.md item 2.4 (Local VM integration testing on ubongo, pre-deploy). Date: 2026-06-18. Implements: the concrete build of ADR-008 Level 2/3 (staging/integration), deferred for lack of hosts but hostable on ubongo. To be recorded as ADR-025.

Context

Molecule (ADR-008 Level 1) tests each role in a single Docker container: one converge, no real kernel netfilter, no real Docker daemon in the loop, and no reboot. That structurally cannot catch an entire class of bug — reboot-survivability, host-firewall × Docker interaction, and boot-ordering — which is exactly the class that caused the 2026-06-17 mesh-hardening incident:

• base's nftables forward { policy drop; } killed the askari Docker host on reboot (nftables loaded its default-deny before Docker, breaking published-port DNAT and inter-container forwarding → public services + the mesh went down). It had worked right after make deploy, when Docker's runtime rules still coexisted. (FRICTION 2026-06-17 #1.)
• ip_nonlocal_bind did not beat the sshd boot-race; sshd bound to the wt0 mesh IP had no listener at boot. (FRICTION #2.)
• The coordinator host could not bootstrap the mesh it itself hosts. (FRICTION #3.)
• NetBird netbird-server FATAL-loops on the GeoLite2 download when egress is lost — and egress was lost when nft flush wiped Docker's NAT masquerade. (FRICTION #4.)

Recovery needed the Hetzner console + a WAN-SSH break-glass. The lesson, already crystallised as a standing rule: firewall/sshd/boot changes must be tested on a real VM with a real reboot before they touch a live host, and a non-mesh break-glass must be kept.

This spec defines a way for the agent to spin up throwaway KVM VMs locally on ubongo that mirror a target host (real Docker, a real reboot, the real role apply) and validate risky infra changes before a live deploy. ubongo can host this today:

verified: ubongo KVM capability · Bash (this session) · /dev/kvm present + accessible (kvm group), Intel VT-x (vmx) enabled, 8 vCPU (i3-10100T), ~13 GiB RAM free of 16, ~198 GiB disk free; libvirt/QEMU/Vagrant not yet installed · 2026-06-18.

Goals

• Reproduce the 2026-06-17 bug class locally: real OS boot, real Docker, real netfilter, the real role apply, a real reboot, then outcome assertions.
• Let the agent drive the full loop autonomously: provision → apply → reboot → assert → teardown, with diagnostics captured on failure.
• Mirror a real host from inventory (first profile: "be askari"), so the apply is faithful, not synthetic.
• Be the concrete tool that operationalises the standing "test risky infra before live deploy" rule.

Non-goals (v1)

• Not a production hypervisor on ubongo (reconciles ADR-015 — see Governance).
• Not nested Proxmox; the provisioning chrome (template clone / Terraform) is not mirrored — every incident bug lives in the boot/kernel/Docker layer, not provisioning.
• Not a multi-VM mini-cluster; one VM at a time. (All six 2026-06-17 signals occurred on a single host that was Docker host + coordinator + mesh peer.) Multi-VM is a later extension.
• Not a CI gate; this is an interactive, agent-driven pre-deploy check on ubongo (CI stays lint + Molecule per ADR-008/010).

Decisions (from the 2026-06-18 brainstorm)

1. Virtualisation approach: libvirt/KVM directly (Approach A). A golden Debian-13 genericcloud qcow2 cached locally; each run boots an ephemeral qcow2 overlay backed by it, seeded via cloud-init NoCloud, driven by a stdlib-only Python script over virsh (no libvirt-python dependency). Chosen over Vagrant+vagrant-libvirt (Ruby/plugin footprint, box drift from the real cloud image) and terraform-provider-libvirt (poor at the imperative apply→reboot→re-apply sequence, throwaway state, blurs ADR-006's prod-VM boundary). Lightest footprint on a 15 GiB control node; full control of the reboot step; the same Debian cloud image real hosts boot.

2. Fidelity envelope: real OS/Docker/netfilter/reboot, not the Proxmox provisioning path. A lightweight local hypervisor is enough because the bugs are post-boot.

3. Scope: one throwaway VM at a time, instantiated from a real host's inventory. First profile: "be askari" (Docker host + NetBird coordinator + mesh peer on one box). The mechanism is generic — later "be" any host by swapping which inventory host it mirrors.

4. Acceptance is self-validating against the real failure. Done = the harness, on a local VM, applies base (firewall on) to a Docker host, reboots, and observes the 2026-06-17 breakage (Docker forwarding dead / services down); then, with the docker_host container-forward drop-in in place, the same run survives the reboot. If step 1 passes, the harness is not faithful.

5. Tiered cert fidelity via a --certs knob (DNS-01 is what makes real certs possible with no public inbound — validation is out-of-band via a Gandi TXT record; the VM needs only outbound to ACME + Gandi, which the NAT net provides):

   • internal (default) — Caddy tls internal, zero deps, instant; for the incident repro and runs where certs aren't under test.
   • le-staging — real DNS-01 ACME against Let's Encrypt staging: real caddy-gandi path, real cert files/renewal, untrusted root, effectively no rate limits. Built in v1.
   • le-prod-wildcard — a real trusted *.test.wingu.me wildcard, issued once, persisted on ubongo, reused across runs. Wired in v1 but on-demand only; its accepted risk is recorded when used (prod Gandi credential reaching an ephemeral VM; transient TXT in the real wingu.me zone). A deliberate "no-egress" failure scenario (to reproduce FRICTION #4) forces internal, since ACME needs egress.

6. The toolchain is Ansible-managed, not hand-installed: a new non-service role (integration_test, control group) installs/enables libvirt+QEMU reproducibly. The repo owns ubongo's state. The driver manages images lazily on first run (keeps the role lean; avoids fiddly download/refresh logic in Ansible).

7. Stubs live in an overlay file, never in the real inventory — so make tf-inventory and "don't edit inventory directly" stay intact, and every stub is explicit and reviewable.

8. A new ADR-025 records this decision (approach + alternatives + cert tiers); ADR-008 gains a pointer and redirects its "what Molecule does NOT test" gaps here.

Architecture — five isolated components

#	Component	Purpose	Location
1	integration_test role (non-service, control group)	Install/enable libvirt+QEMU+virtinst, add sjat/claude to libvirt group, create the image-cache dir, drop the driver. Idempotent, Molecule-tested.	roles/integration_test/
2	integration-vm.py driver	Stdlib-only lifecycle over virsh: up / apply / reboot / assert / cycle / reset / down / prune / console. Lazily ensures the golden image (download + checksum).	scripts/integration-vm.py
3	Profiles + var overlays	Make a VM "become" a host: pull that host's real group_vars/host_vars + layer a small explicit overlay (cert tier, in-VM coordinator endpoint, VM connection).	tests/integration/overrides/<host>.yml
4	Verify playbook	Outcome-based post-reboot assertions (Docker up, published-port DNAT alive, nft sane, service responds, wt0 up), reusing Molecule's verify.yml philosophy.	tests/integration/verify.yml
5	Makefile target	make test-integration HOST=<name> [CERTS=...] [KEEP=1] → cycle; make test-integration-clean → prune. Documented in CLAUDE.md's command table.	Makefile

Lifecycle / data flow

make test-integration HOST=askari drives:

 1. ensure golden image    Debian-13 genericcloud qcow2, cached + checksum-verified
 2. ephemeral overlay      qcow2 backed by golden (throwaway; never mutate golden)
 3. cloud-init NoCloud      seed hostname + ansible user + ubongo's SSH key + NIC
 4. virt-install --import   boot on an isolated libvirt NAT net (DHCP IP + outbound NAT)
 5. wait for SSH            IP via `virsh domifaddr --source lease` (guest-agent optional)
 6. transient inventory     askari's real vars + ansible_host=<lease IP> + stub overlay
 7. ansible-playbook site   THE REAL APPLY (base + docker_host + reverse_proxy + coordinator)
 8. [snapshot post-apply]   optional reset point for fast re-runs
 9. virsh reboot ──────────┐  ← the step Molecule structurally cannot do
10. wait for SSH           ┘
11. ansible-playbook verify outcome assertions; THIS is where the incident surfaces
12. report + teardown       pass/fail; on fail keep VM + dump diagnostics; else destroy overlay

Steps 1–7 build a real Docker daemon with real published-port DNAT to break; step 9 is a real kernel reboot, so nftables loads default-deny before Docker exactly as on askari.

Fidelity boundary & cert tiers

Faithful where the bug lives: real kernel, real netfilter, real Docker with published-port DNAT, the real role apply, a real reboot, and the coordinator running inside the VM so the VM is its own mesh peer — reproducing the circular mesh-bootstrap (FRICTION #3) on one box.

Stubbed where it needs the public internet (explicit, in the overlay): LE certs via the --certs knob (Decision 5); public DNS (askari.wingu.me) → local resolution; NetBird geo-DB → pre-seeded or requirement disabled (which is also the FRICTION #4 fix, so the harness can test both the FATAL-loop and its remedy).

Acceptance test (self-validating)

1. Run the cycle on today's base (firewall on, no docker_host container-forward drop-in) → step 11 must FAIL after reboot (Docker forwarding dead, services down).
2. Implement the docker_host container-forward rules (the pending fix STATUS.md names) → re-run → step 11 must PASS across the reboot.

Scope boundary: the harness is this plan's deliverable. The docker_host container-forward fix is a separate work item (FRICTION 2026-06-17 #1). v1's acceptance deliberately spans both, because a credible harness must demonstrate both a true-negative (red on the broken state) and a true-positive (green on the fixed state) — otherwise we have only ever watched the assert go red. The plan decides sequencing: build the small docker_host drop-in as the green-half of acceptance, or consume it if built separately first. Minimum credible v1 is the red half (faithful reproduction); full acceptance is red→green.

This one round-trip proves the harness reproduces the incident, the fix works, and the loop can be trusted for the next risky change before it touches a live host.

Robustness, isolation & teardown

Failure leaves evidence (catching a bug is the point):

Step fails	Behaviour
Golden image (1)	Fail fast, clear message; image cached (one-time cost)
Boot / first SSH (4–5)	Capture serial console to a log file, fail with its tail — the automated equivalent of the Hetzner console (ties to TODO 10.8)
Apply (7)	Keep VM, surface Ansible output, dump diagnostics
No SSH after reboot (9–10)	The classic incident signature; FAIL, keep VM, capture console — the harness succeeding
Assert (11)	FAIL, keep VM, dump post-mortem: nft list ruleset, docker ps, ss -tlnp, journalctl -b, systemd-analyze critical-chain; exit non-zero

Diagnostics land in gitignored ~/integration-runs/<ts>-<host>/ (same pattern as ADR-017's screenshot dir; the agent reads them directly).

Three safety invariants (these make the testing tool itself safe):

1. The transient inventory contains only the test VM — no real host is ever in scope; the apply is --limited to the VM.
2. "Be askari" points NetBird at the in-VM coordinator (localhost) — the VM forms its own one-node mesh; it never enrolls in the real mesh.
3. Test VMs sit on an isolated libvirt NAT net — outbound NAT for ACME/image pulls, but not reachable to the LAN (10.20.x) or the real mesh.

Resource guard (ubongo's 15 GiB ceiling, ADR-015/012): default VM ≈ 2 vCPU / 3 GiB / 20 GiB thin overlay; the driver refuses to start below a free-RAM threshold and enforces one integration VM at a time (name-prefix boma-it-*). Teardown: success destroys domain + overlay; failure keeps them and prints how to inspect; make test-integration-clean reaps all boma-it-* orphans. An optional post-apply snapshot lets reset re-run reboot+assert without re-applying (fast iteration on a fix).

Testing the tester

• pytest on the driver's pure logic: transient-inventory generation, var/overlay merge, --certs→overlay mapping, DHCP-lease parsing, resource-guard math (mock virsh). Joins boma's existing pytest suite.
• Molecule (Docker) on the integration_test role: asserts libvirt/qemu/virtinst installed, libvirtd enabled, users in libvirt group, driver present. (Cannot run KVM-in-Docker — the documented Molecule limitation.)
• End-to-end self-test = the acceptance test above, run manually on first build and recorded in the runbook.

Governance & documentation touch-points

• ADR-025 "Local VM integration testing" — decision, approach A, rejected alternatives (Proxmox-nested / Vagrant / TF-libvirt), cert tiers.
• ADR-008 — pointer to ADR-025; redirect its "what Molecule does NOT test" gaps (nftables loading, mesh dataplane) to this level.
• ADR-015 — one-line reconciliation: "not a hypervisor" → runs ephemeral KVM test VMs as part of its local-test-runner role (still not a production hypervisor); note the test-VM RAM load.
• docs/security/accepted-risks.md — the le-prod-wildcard risk (prod Gandi credential → ephemeral VM; transient TXT in real wingu.me).
• CLAUDE.md command table + docs/runbooks/integration-testing.md (run a cycle, cert knobs, where diagnostics land, inspecting a kept failed VM, pruning) + STATUS.md entry. The runbook's pre-flight line operationalises FRICTION #6 (validate reboot-recovery before retiring the break-glass).

Capacity

One VM (~3 GiB) against ~13 GiB free is comfortable. The only future pinch is concurrency with the Level-4 Chromium/Playwright stack (ADR-017) — handled by the resource guard + "one at a time." Add a note to docs/hardware/reference.md; revisit at /capacity-review.

Alternatives considered

• Proxmox VE nested on ubongo — highest fidelity incl. the provisioning step, but heavy (nested virt, RAM), in tension with ADR-015, and the incident bugs don't live in provisioning. Rejected.
• Vagrant + vagrant-libvirt — mature lifecycle/snapshots, but adds the Ruby/Vagrant ecosystem + a fragile plugin, boxes drift from the real Debian cloud image, and the reboot→assert sequence still needs custom logic. Rejected.
• terraform-provider-libvirt — declarative and reuses TF, but poor at the imperative apply→reboot→re-apply test sequence, adds throwaway state, and blurs ADR-006's "TF owns production VM existence on Proxmox" boundary. Rejected.

Open questions / deferred

• Multi-VM mini-staging (inter-host mesh/dataplane) — design the driver + NAT net so a topology is an additive extension; out of scope for v1.
• Interplay with the Level-4 browser stack — both want ubongo RAM; the resource guard is the v1 answer, revisit when Level 4 is built.
• Snapshot strategy depth — v1 ships clone-and-destroy + an optional post-apply snapshot; richer snapshot trees deferred.

Knowledge to verify at plan stage (ADR-014)

These are from memory / unverified and must be confirmed against version-matched docs before the plan asserts them:

• Exact virt-install --import flags and the cloud-init NoCloud seed format on the Debian-13 libvirt stack.
• Whether the Debian-13 genericcloud image ships qemu-guest-agent (IP can come from the DHCP lease regardless — guest-agent is an optimisation, not a requirement).
• Let's Encrypt rate limits (prod vs staging) — to confirm "issue the wildcard once, reuse" stays within limits.
• The caddy-dns/gandi DNS-01 configuration and pinned version already used by reverse_proxy, and whether the Gandi LiveDNS API key can be scoped to test.wingu.me.
• libvirt default vs a dedicated isolated NAT network on Debian-13 (virsh net-*).