boma/docs/superpowers/specs/2026-06-19-mesh-hardening-ubongo-default-deny-design.md

Spec — Mesh-hardening (2 of 3): ubongo INPUT-only default-deny + ssh-from-control

Status: Accepted (2026-06-19)

Context & scope

The mesh-hardening follow-on (deferred from M5, ROADMAP) was decomposed into three independent sub-projects, each its own spec → plan → implementation cycle:

1. askari SSH → wt0 — spec/plan written 2026-06-17, attempted and backed out the same day (the incident; six lessons in FRICTION.md). Needs a redesign — not this spec.
2. ubongo nftables default-deny + ssh-from-control ← this spec
3. NetBird ACL off Allow-All → scoped policies (its own later spec; open mechanism question — no headless API path).

ROADMAP (re-ordered after the 2026-06-17 incident) puts ubongo first: it is the clean, low-risk case — a physical box with a permanent console break-glass, and not the coordinator host that the incident proved you must not corner.

This spec hardens ubongo's inbound surface only. It does not change sshd's ListenAddress (so no boot-race), does not apply a forward-chain default-deny (so Docker + the libvirt NAT keep working), and does not touch askari or the NetBird ACL.

Current state (verified on ubongo, 2026-06-19): no host firewall — sshd listens on 0.0.0.0:22, reachable from LAN, mesh, and anything routable; only Docker's + libvirt's own iptables-nft tables exist. Interfaces: eno1 10.20.10.151 (LAN, = ansible_host), wt0 100.99.146.14 (mesh), docker0 (one container, no published ports), virbr-boma 192.168.150.1/24 (the libvirt NAT that make test-integration uses), ip_forward=1.

Goal / success criteria

• SSH to ubongo succeeds over wt0 (road-warriors, askari), from mamba on the LAN (10.20.10.50), and via the ssh-from-control self-path (Ansible; source 10.20.10.151).
• SSH from any other LAN source is dropped (default-deny on input).
• Docker container egress and make test-integration (libvirt NAT) keep working — the forward chain is untouched.
• A reboot does not lock SSH out (no ListenAddress, so no bind race).
• Break-glass is the on-prem physical console (permanent, non-mesh). The live apply is additionally gated by the firewall auto-rollback timer.

Design

Apply base's nftables firewall concern to ubongo, with two adjustments and one deliberate non-change:

1. INPUT-only default-deny. The input chain keeps policy drop with the guaranteed management plane: lo, established,related, ICMP, SSH on wt0, and SSH from ssh-from-control (10.20.10.151). We add one operator-workstation source (mamba, 10.20.10.50) via a new base__firewall_admin_addrs list. Everything else on eno1 drops.
2. Forward chain left permissive. base hardcodes chain forward { … policy drop; } for inter-container isolation. On ubongo that would break Docker egress and the libvirt NAT the integration harness depends on — the same class of failure that sank askari (FRICTION 2026-06-17, signal 1). A new base__firewall_input_only knob renders the forward chain policy accept instead. Docker's and libvirt's own iptables-nft forward rules continue to apply (separate tables); base simply does not add a default-deny on top.
3. No sshd ListenAddress change. sshd keeps listening on 0.0.0.0:22; nftables does all inbound scoping. This deliberately avoids the ip_nonlocal_bind boot-race that broke askari (FRICTION signal 2) — there is nothing to bind before wt0 exists.

Resulting input allow-list:

iif "lo" accept
ct state established,related accept
ct state invalid drop
iifname "wt0" tcp dport 22 accept            # mesh (road-warriors, askari)
ip saddr 10.20.10.151 tcp dport 22 accept    # ssh-from-control (Ansible self) — group_vars/all
ip saddr 10.20.10.50  tcp dport 22 accept    # mamba on the LAN — base__firewall_admin_addrs
ip protocol icmp accept ; ip6 nexthdr ipv6-icmp accept
# (no catalog services on ubongo) → default drop
chain forward: policy accept                 # Docker + libvirt-NAT forwarding preserved

Why ubongo is the safe case (maps to the 2026-06-17 incident)

• Signal 1 (forward-drop breaks Docker hosts): sidestepped — INPUT-only leaves forwarding alone.
• Signal 2 (ip_nonlocal_bind boot-race): sidestepped — no ListenAddress; sshd binds nothing new.
• Signal 3 (a host's only mgmt path must not depend on a service it hosts): satisfied — ubongo is not the coordinator and keeps three independent paths (mesh, LAN, physical console).
• Signal 6 (recovery tested after the break-glass was removed): the physical console is permanent (nothing to retire), and reboot-recovery is proven on a throwaway VM first.

New & changed code

Role base:

• roles/base/defaults/main.yml — add:
  • base__firewall_input_only: false — when true, the forward chain is policy accept (host-local input filtering only), for hosts that route/forward container or NAT traffic (e.g. the control node's Docker + libvirt-NAT) where a forward default-deny would break them.
  • base__firewall_admin_addrs: [] — extra LAN source IPs allowed to SSH (besides wt0 + ssh-from-control); for an operator workstation reaching the host over the LAN. Key-gated.
• roles/base/templates/nftables.conf.j2:
  • the forward line (currently line 21) → chain forward { type filter hook forward priority 0; policy {{ "accept" if base__firewall_input_only | bool else "drop" }}; }
  • after the ssh-from-control block (currently lines 12-14), add a loop: {% for addr in base__firewall_admin_addrs %} → ip saddr {{ addr }} tcp dport {{ base__firewall_ssh_port }} accept
• roles/base/molecule/default/{converge,verify}.yml — fixture sets input_only: true + an admin_addrs entry; assert (a) forward renders policy accept, (b) the admin-addr accept rule renders, (c) existing input default-deny + wt0 + control-addr assertions stay green.

Inventory (inventories/production/group_vars/control/vars.yml, append):

# Mesh-hardening 2/3 (2026-06-19, ADR-020/021): apply base's host firewall to ubongo as
# INPUT-only default-deny — harden the inbound surface, leave the forward chain permissive so
# Docker egress + the libvirt-NAT integration harness keep working. sshd is unchanged
# (nftables scopes inbound), so there is no boot-race. Reach ubongo over wt0, the
# ssh-from-control self-path (base__firewall_control_addr in group_vars/all), or mamba on the
# LAN. Break-glass: the physical console.
base__firewall_input_only: true
base__firewall_admin_addrs:
  - "10.20.10.50"   # mamba over the LAN (NetBird off). Raw DHCP lease — see note below.
# base__firewall_apply defaults true; base__firewall_control_addr (= ubongo's own 10.20.10.151)
# is set in group_vars/all and covers Ansible's self-connection.

Integration harness (ADR-025) — a "be ubongo" profile, mirroring "be askari":

• tests/integration/overrides/ubongo.yml — firewall_apply: true, input_only: true, admin_addrs: ["192.168.150.99"] (a representative LAN addr to exercise the rule), firewall_control_addr: "192.168.150.1" (the libvirt-NAT gateway = the harness's own SSH path, so the apply + reboot don't lock it out), ssh_listen_mesh_only: false, mesh_enabled: false.
• tests/integration/profiles/ubongo.json — mirror profiles/askari.json (VM resources/image).
• tests/integration/verify.yml — make the assertions profile-aware (gated on the active profile, since verify.yml is shared): for ubongo assert input policy drop, forward policy accept, and the admin-addr rule present. Reachability across the reboot is the harness's existing cycle. The askari assertions (Docker/forward-DNAT) must not run for the ubongo profile, nor vice-versa.

Enables make test-integration HOST=ubongo.

The mamba admin-addr — a deliberately interim value

base__firewall_admin_addrs: ["10.20.10.50"] is mamba's current raw DHCP lease, not a reservation (operator decision, 2026-06-19). Caveats, accepted for now:

• Lease drift: if DHCP reassigns 10.20.10.50, the rule allows whatever host then holds it (still SSH-key-gated, so low risk) and mamba loses its LAN path. Backstop: mamba also reaches ubongo over wt0 (mesh), so it is never cut off — only the off-mesh LAN convenience lapses until the IP is corrected.
• Revisit trigger: when OPNsense-as-code lands (ADR-020 perimeter layer), replace this with a DHCP reservation (MAC → fixed IP) and allow the reserved address. Tracked here and in the implementation plan's follow-ups.

Testing

• Molecule (base default, render-only, firewall_apply: false): the new forward-accept + admin-addr assertions above, with existing assertions green.
• Integration harness (make test-integration HOST=ubongo): on a throwaway UEFI VM, apply the ubongo overlay, assert the ruleset shape, and prove SSH survives a reboot from an allowed source (the existing assert/cycle). This is the gate before touching the real control node.
• Live (during cutover): SSH over wt0 ✓, from mamba LAN ✓, Ansible self-ping ✓; SSH from a disallowed LAN host dropped ✓; docker run … egress ✓; a fresh make test-integration still spins a VM (libvirt NAT intact) ✓.

Staged cutover (operator-supervised — lockout-aware, FRICTION signal-6 order)

ubongo is managed as sjat (password sudo), so the live apply needs the operator present anyway. The physical console is open throughout.

1. Harness GREEN: make test-integration HOST=ubongo passes (incl. the reboot).
2. Pre-check the real paths before applying: SSH over wt0, SSH from mamba (10.20.10.50), ansible ubongo -m ping. Confirm the physical console is reachable.
3. Dry-run: make check PLAYBOOK=site LIMIT=ubongo TAGS=firewall — review the nftables diff (input default-deny + wt0 + 10.20.10.151 + 10.20.10.50; forward policy accept).
4. Apply (auto-rollback armed): make deploy PLAYBOOK=site LIMIT=ubongo TAGS=firewall — the firewall concern snapshots, arms the 45 s revert, applies, reset_connection → wait_for_connection over the live path (10.20.10.151), then cancels the timer. A bad ruleset reverts itself; the console is the ultimate fallback.
5. Verify every path + Docker egress + a fresh integration-VM spin (above).
6. Reboot ubongo; confirm SSH returns on all paths unaided (console present). Only now is it done — recovery is proven while the break-glass is still there.
7. Docs: update STATUS.md (ubongo row: input-only default-deny applied) and ROADMAP.md (mesh-hardening 2/3 done; next is sub-project 1 askari redesign or 3 NetBird ACL).

Risks & rollback

• Self-referential apply (ubongo runs Ansible against itself): mitigated by the auto-rollback timer, the wait_for_connection over the real path, three redundant allowed sources, and the permanent physical console. ubongo cannot be bricked.
• Raw-lease fragility: documented above; backstopped by the mesh path; revisit with OPNsense.
• No new container isolation (forward stays accept): accepted — ubongo is a single-tenant control node, not a service host; Docker/libvirt keep their own forward rules. The forward default-deny remains the norm for real service hosts (base__firewall_input_only: false).

Out of scope / follow-ons

• askari SSH → wt0 redesign (sub-project 1) — needs the boot-race + coordinator-bootstrap resolved; folds in the coordinator-robustness (geo-DB FATAL-loop) + off-site backup lessons.
• NetBird ACL off Allow-All (sub-project 3) — open mechanism question (no headless API path).
• OPNsense DHCP reservation for mamba (and ubongo) — replaces the raw lease; with OPNsense-as-code.
• Forward-chain container isolation on ubongo — deliberately not done here.
• STATUS.md / ROADMAP.md edits land with the implementation, not this spec.