boma/docs/superpowers/specs/2026-06-20-mesh-spof-accept-resilience-design.md

# Spec — Mesh-hardening (SPOF): accept the single-coordinator SPOF + targeted resilience

Status: Accepted (2026-06-20)

## Context & scope

The **mesh-hardening follow-on** decomposed into independent sub-projects (ROADMAP). Progress:

1. ~~ubongo nftables INPUT-only default-deny~~ — **DONE 2026-06-19**.
2. ~~askari SSH → `wt0` redesign~~ — **DONE 2026-06-20** (live reboot-validated).
3. **askari relay-SPOF reduction** ← *this spec*.
4. NetBird ACL off Allow-All — not started.

`askari` runs boma's **single** self-hosted NetBird coordinator (management + signal + relay +
STUN, one combined container) **and** is a mesh peer (ADR-016). Because `ubongo`'s INPUT-only
default-deny drops the inbound UDP that ICE hole-punching needs, `ubongo`'s peers are always
**`Relayed`** through askari's own relay (intentional posture — `docs/runbooks/netbird-client.md`,
the `ubongo-relay-only` finding). So askari is a single point of failure for **relayed mesh
traffic**.

### The decisive finding — the blast radius is narrow

The mesh (`wt0`) is **not** a default gateway. Verified on ubongo (2026-06-20):

```
wt0 routes ONLY 100.99.0.0/16   ·   default route via 10.20.10.1 dev eno1   ·   Networks: -  (no subnet-routes/exit-node)
```

So an askari outage affects **only** traffic addressed to a peer's `100.99.x.x` mesh IP over the
relay:

| Traffic | askari down |
|---|---|
| LAN device → LAN service (direct or via reverse proxy) | unaffected |
| node ↔ node over LAN IPs (future cluster) | unaffected |
| node ↔ node same-LAN over mesh IPs | unaffected (direct P2P, local ICE candidate) |
| **road-warrior → ubongo (remote, relayed)** | **breaks** |
| mesh control plane (new enrol / ACL change / re-handshake) | pauses |

Nothing on the LAN and no future intra-cluster traffic depends on askari. The only loss is
**remote (off-LAN) mesh access to peers** — and only when off-LAN *and* askari is down at once.

### Why we are not "fixing" the SPOF with new infrastructure

- **A second coordinator** is not supported by self-hosted NetBird (single management/signal) and
  contradicts ADR-016's deliberate single off-site coordinator.
- **Direct P2P** only helps already-established sessions (re-handshakes still need askari's
  signal), and enabling it punctures `ubongo`'s deliberate default-deny (a firewall-catalog UDP
  entry + an `accepted-risks` deviation + OPNsense NAT) — cost out of proportion to a narrow,
  rare failure.
- **A second relay** needs another publicly-reachable host; a relay at home reintroduces the
  public home surface ADR-016's off-site coordinator exists to avoid.

Given a reliable always-on VPS and boma's 2–5-host scale, the sound engineering choice is to
**accept the SPOF as a conscious, documented trade-off** and harden only the two spots real
incidents point to.

## Goal / success criteria

- The single-coordinator SPOF is **explicitly accepted and documented** (register entry + an
  ADR-016 availability analysis + recovery), so the trade-off is revisitable, not forgotten.
- **Managed mesh hosts survive a local-DNS hiccup:** `ubongo` (and future managed mesh hosts)
  resolve the coordinator FQDN even when their resolver dies on a transition, mirroring the
  client-side fix already in the runbook.
- **No new infrastructure** — no P2P, no second relay, no second coordinator, no Terraform.
- The coordinator **off-site backup gap** is named in the accepted risk and explicitly handed to
  the next sub-project (ADR-022), not built here.

## Design

### (a) Accepted-risk `R8` — `docs/security/accepted-risks.md`

Add one row to the register (owned by ADR-002):

- **Risk:** *Single off-site mesh coordinator is an availability SPOF for remote mesh access* —
  askari hosts the only management/signal/relay (ADR-016); a relayed peer (all of ubongo's) loses
  remote mesh reachability while askari is down, and the control plane pauses. The
  `netbird_coordinator` store has **no off-site backup yet** (BACKUP.md), so an askari loss also
  loses mesh control-plane state until rebuilt.
- **Rationale:** inherent to ADR-016's deliberate single off-site coordinator (sovereignty,
  survives a homelab outage); **narrow blast radius** (above table — LAN/intra-cluster/local
  unaffected); askari is a reliable always-on VPS; mitigations exist (client + managed-host DNS
  pin; documented rebuild).
- **Revisit trigger:** askari proves unreliable; the cluster grows to depend on the mesh for
  intra-node traffic; remote mesh access becomes business-critical; or the ADR-022 backup role
  lands (closes the state-loss half).

R8 is the **availability** complement to R3 (which covers askari as a *security* target).

### (b) ADR-016 amendment — an "Availability — an askari outage" subsection

A short subsection capturing: the blast-radius table; that the SPOF is an accepted property
(→ R8); and the **recovery procedure** — rebuild the coordinator (`/setup` + re-enrol peers, M5)
or restore from backup once ADR-022 lands; client/road-warrior break-glass already in
`docs/runbooks/netbird-client.md`; on-LAN access to ubongo never depends on the mesh (ADR-016
recovery model). Recorded as an amendment (dated), ADR-016 status stays Accepted.

### (c) DNS-resilience — pin the coordinator FQDN on managed mesh hosts (`base` `mesh` concern)

The 2026-06-18 outage was a client failing to resolve `netbird.askari.wingu.me` on a network
transition; the client fix (public resolvers + an `/etc/hosts` pin to askari's stable WAN IP) is
already in the runbook. The gap: **managed** mesh hosts have no equivalent. Add to `base`'s `mesh`
concern (`roles/base/tasks/mesh.yml`):

- New default `base__mesh_coordinator_pin: ""` (empty → no pin; opt-in).
- When set (and `base__mesh_enabled`), render an `/etc/hosts` entry mapping the coordinator FQDN
  — derived from `base__mesh_management_url` via the `urlsplit('hostname')` filter, **not** a
  duplicated literal — to `base__mesh_coordinator_pin`, idempotently (a marker-scoped
  `blockinfile`/`lineinfile`).
- Set `base__mesh_coordinator_pin` to askari's static WAN IP for managed mesh hosts that depend
  on the coordinator (ubongo via the `control` group_vars; future cluster groups as they appear).
  The **coordinator host itself (askari) is exempt** (it would point its own FQDN at its own WAN
  IP — needs NAT hairpin and is a server with stable DNS); the plan confirms the exact group_vars
  placement and the askari exemption.

The pin is safe because askari's WAN IP is static (operator-confirmed); rendering it from a single
inventory variable keeps it maintainable if it ever changes.

## New & changed code/docs

- `docs/security/accepted-risks.md` — add row **R8**; bump the "Last reviewed" date.
- `docs/decisions/016-mesh-vpn.md` — add the dated "Availability — an askari outage" amendment
  subsection (blast-radius table + recovery + R8 cross-ref).
- `roles/base/defaults/main.yml` — add `base__mesh_coordinator_pin: ""` with a comment.
- `roles/base/tasks/mesh.yml` — add the `/etc/hosts` coordinator-pin task (gated on
  `base__mesh_enabled` + a non-empty pin; FQDN from `urlsplit`).
- `inventories/production/group_vars/control/vars.yml` — set `base__mesh_coordinator_pin` to
  askari's WAN IP for ubongo.
- `roles/base/molecule/default/{converge,verify}.yml` — assert that with the pin set + a fixture
  FQDN the `/etc/hosts` entry renders, and that an empty pin renders nothing (no-op).
- `STATUS.md` / `docs/ROADMAP.md` — mark sub-project 3 done; surface ADR-022 (coordinator backup)
  as the next item. (Land with the implementation, not this spec.)

## Testing

- **Molecule** (`base` default scenario): (1) `base__mesh_coordinator_pin: ""` → no `/etc/hosts`
  coordinator line (default no-op); (2) pin set + a fixture `base__mesh_management_url` → exactly
  one idempotent `<ip> <fqdn>` line, FQDN correctly extracted by `urlsplit`. Existing
  firewall/hardening/mesh assertions stay green.
- **No live deploy required for acceptance** — the pin is additive and idempotent; it lands on
  ubongo on the next routine `base` apply. (Optional spot-check: `getent hosts
  netbird.askari.wingu.me` on ubongo resolves to the pinned IP.)

## Risks & rollback

- **Stale pin if askari's WAN IP changes** — mitigated by rendering from one inventory variable
  (single edit) and askari's IP being static; the pin is removable by clearing the knob + a
  re-apply.
- **Over-pinning the coordinator host** — askari is explicitly exempt (hairpin/DNS), set in
  group_vars scope.
- **Accepting the SPOF** is itself the residual risk — bounded by the narrow blast radius, the
  documented recovery, and R8's revisit triggers.

## Out of scope / follow-ons

- **Coordinator off-site backup → ADR-022 kickoff (the next sub-project).** Named in R8 and
  `BACKUP.md` as the open gap; building it means ADR-022's pull-node (`fisi`) + restic design, not
  throwaway plumbing here.
- **Direct P2P / NAT-traversal** — deferred posture change (default-deny puncture + OPNsense NAT +
  governance); explicitly not pursued here.
- **A second relay / second coordinator** — ruled out above (infra cost / not supported / against
  ADR-016).
- **NetBird ACL off Allow-All** — separate sub-project (4).