boma/docs/superpowers/specs/2026-06-17-mesh-hardening-askari-ssh-wt0-design.md

Spec — Mesh-hardening (1 of 3): move askari's SSH onto wt0

Status: Accepted (2026-06-17)

Context & scope

The mesh-hardening follow-on was deferred from M5 (ROADMAP). It was decomposed into three independent sub-projects, each with its own spec → plan → implementation cycle:

1. askari SSH → wt0 ← this spec
2. ubongo nftables default-deny + ssh-from-control (its own later spec)
3. NetBird ACL off Allow-All → scoped policies (its own later spec)

This spec covers only (1). It makes askari's SSH reachable only over the NetBird mesh interface wt0 and closes the WAN :22 surface at both the host and the Hetzner Cloud Firewall. It does not touch ubongo, the NetBird ACL (stays Allow-All for now — one moving access-layer at a time), or askari's public service exposure (Caddy 80/443, NetBird STUN 3478 stay on the WAN).

Current state (STATUS.md): askari is reached at ansible_host: 77.42.120.136 (WAN, in the TF-generated inventories/production/offsite.yml); wt0 is up at 100.99.226.39 (Management+Signal Connected, M5); the base nftables firewall concern is built but not applied to askari (the Hetzner Cloud Firewall is its perimeter today); the Hetzner Cloud Firewall (terraform/modules/hetzner_vm) opens :22 from var.ssh_admin_cidrs plus 80/443/3478 from anywhere.

Goal / success criteria

• SSH to askari succeeds over wt0 (from ubongo) and fails from any off-mesh source.
• The WAN :22 surface is closed at both layers (host nftables = wt0-only; Hetzner Cloud Firewall drops the :22 rule).
• Public services are unaffected: https://test.askari.wingu.me and https://netbird.askari.wingu.me serve valid certs; STUN 3478/udp still answers.
• Ansible manages askari over wt0.
• Break-glass is the Hetzner web console (out-of-band; works even if the mesh is down).
• A reboot of askari does not lock SSH out (the boot-race below is solved).

Design — three enforcement layers (defense-in-depth)

1. sshd binds ListenAddress to askari's wt0 IP only, so it does not accept on WAN.
2. host nftables (base firewall concern, ADR-020): catalog-driven default-deny; :22 allowed only via iifname "wt0" (the interface-name match that survives wt0 being absent — see docs/testing/gotchas.md); public service ports stay open on WAN.
3. Hetzner Cloud Firewall (Terraform): the :22 ssh_admin_cidrs rule is removed; 80/443/3478 stay.

The boot-race fix (load-bearing)

wt0 is brought up by NetBird after boot, so at sshd start the wt0 IP may not exist yet. A plain ListenAddress 100.99.226.39 would fail to bind → sshd exits → lockout on reboot. Solution:

• net.ipv4.ip_nonlocal_bind = 1 via a sysctl drop-in (ansible.posix.sysctl, persisted under /etc/sysctl.d/). This lets sshd bind the wt0 address even before the interface exists; once wt0 comes up with that IP, traffic is delivered to the existing listener — no reload needed.
• The sshd drop-in fails closed: the mesh IP is resolved (see below) and the play asserts it is non-empty before rendering. An empty ListenAddress would silently fall back to listening on all interfaces, defeating the restriction — that must never render.

Mesh-IP source (decided): the live wt0 fact ansible_wt0.ipv4.address, gathered at apply time (wt0 is up during the play, since M5), with a host_var fallback (base__ssh_listen_addr, default "") and a fail-closed assert that one of them yielded a non-empty address. Live fact is preferred (correct even if NetBird reassigns the IP); the host_var is an explicit override / belt.

New & changed code

Role base (the hardening + firewall concerns):

• roles/base/defaults/main.yml — add:
  • base__ssh_listen_mesh_only: false — opt-in; when true, sshd binds the mesh IP only.
  • base__ssh_listen_addr: "" — optional explicit mesh-IP override (fallback to the ansible_wt0 fact).
• roles/base/tasks/ssh.yml —
  • resolve the mesh IP (base__ssh_listen_addr or ansible_wt0.ipv4.address) into a fact;
  • assert it is non-empty when base__ssh_listen_mesh_only;
  • set net.ipv4.ip_nonlocal_bind = 1 (sysctl drop-in) under the same condition.
• roles/base/templates/sshd_hardening.conf.j2 — append a conditional ListenAddress {{ resolved_mesh_ip }} block guarded by base__ssh_listen_mesh_only (unset → unchanged behaviour: listen on all). Keep the existing sshd -t validation.

Inventory:

• inventories/production/host_vars/askari.yml (new) — ansible_host: 100.99.226.39 (overrides the TF-generated offsite.yml; host_vars are not regenerated by tf_to_inventory.py). A header comment explains why.
• inventories/production/group_vars/offsite_hosts/vars.yml — add base__ssh_listen_mesh_only: true; ensure base__firewall_apply: true. (base__mesh_enabled is already true for askari — set in M5 — and is a precondition, not a change here.)

Firewall catalog (inventories/production/group_vars/all/firewall.yml):

• Enumerate askari's required ingress so catalog-driven default-deny does not drop a live public service. Derived from the existing reverse_proxy + netbird_coordinator definitions: :22/tcp on the mesh zone (wt0); 80,443/tcp + 3478/udp on the public zone (WAN). The exact catalog/zone YAML is finalised in the implementation plan against the resolve_firewall_rules filter's schema.

Terraform (terraform/environments/offsite + terraform/modules/hetzner_vm):

• Remove the WAN :22 ingress rule (e.g. drop ssh_admin_cidrs from the firewall, or set it empty and guard the rule). Keep 80/443/3478. Applied via make tf-plan/apply TF_ENV=offsite (plan shown before apply).

Staged cutover — a working path at every step

1. Pre-check: confirm ssh sjat@100.99.226.39 and an ansible askari -m ping forced over wt0 both succeed before changing anything.
2. Repoint Ansible: add host_vars/askari.yml (ansible_host = wt0 IP); verify ansible askari -m ping runs over the mesh. WAN :22 still open as a fallback here.
3. Apply base (firewall + sshd together): one make deploy PLAYBOOK=site LIMIT=askari converge applies catalog default-deny (:22 on wt0 + public ports) and the sshd ListenAddress=mesh + ip_nonlocal_bind drop-in. The firewall concern's reset_connection → wait_for_connection (now over wt0) plus the armed auto-rollback timer (base__firewall_rollback_timeout, 45 s) is the safety gate — a bad ruleset reverts itself. The sshd reload cannot drop the in-flight wt0 session. Verify the public services still respond.
4. Retire the Hetzner WAN :22: the Terraform change above; make tf-plan TF_ENV=offsite (review) → make tf-apply. Verify: wt0 SSH works; off-mesh nc -vz 77.42.120.136 22 is refused/times out; :443 open; STUN answers.

Testing

• Molecule (base default scenario; wt0 absent in-container, base__firewall_apply: false render-only): assert (a) the rendered nftables allows :22 via iifname "wt0"; (b) with base__ssh_listen_mesh_only: true + a fixture mesh IP, the sshd drop-in renders ListenAddress <ip> and sshd -t passes; (c) with the flag set but no resolvable mesh IP, the play fails closed (the assert); (d) the ip_nonlocal_bind sysctl task is present. Keep existing firewall/hardening assertions green.
• Live, out-of-band: post-cutover, from an off-mesh host nc -vz 77.42.120.136 22 → refused; :443 → open; from ubongo over wt0, SSH + ansible -m ping succeed; reboot askari (Hetzner console) and confirm SSH-over-wt0 returns without intervention.

Risks & rollback

• Mid-cutover lockout: mitigated by the staged order (a path open at each step), the firewall auto-rollback timer, and ansible_host=wt0 so the connectivity confirm tests the real new path.
• Reboot lockout: mitigated by ip_nonlocal_bind (sshd binds wt0 regardless of interface timing) + the fail-closed assert (never silently listen-all).
• Default-deny breaks a public service: mitigated by enumerating all live ingress into the catalog and the §Testing service checks; reversible by re-running with base__firewall_apply: false or widening the catalog.
• Ultimate break-glass: the Hetzner web console (out-of-band). The TF :22 rule is trivially re-addable.

Out of scope / follow-ons

• ubongo default-deny + ssh-from-control (sub-project 2).
• NetBird ACL off Allow-All (sub-project 3) — until then any enrolled peer can reach askari's wt0:22; scoping that is sub-project 3's job.
• /check-access (ADR-021) live verification — designed, build still pending.
• STATUS.md / ROADMAP updates land with the implementation, not this spec.