boma/docs/superpowers/plans/2026-06-05-mesh-vpn-netbird.md

Mesh VPN (NetBird) Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Record the decision that boma's mesh VPN is NetBird (self-hosted on askari), by authoring ADR-016 and reconciling every doc that currently assumes OPNsense WireGuard or an undecided VPN.

Architecture: Documentation-only change. NetBird replaces ADR-007's VLAN-99 OPNsense WireGuard as the single remote-access overlay for ubongo, askari, and road-warrior clients; coordinator self-hosted off-site on askari; agent-per-host enrollment via the (unbuilt) base role; embedded local-user identity. The role/service implementation waits on the base role and service-role machinery that STATUS.md lists as not-yet-built — this plan settles the decision and the doc reconciliation only.

Tech Stack: Markdown only. Verification is the repo's pre-commit hooks (trailing-whitespace, end-of-file, gitleaks, ansible-lint, vault-encryption guard) plus a final cross-reference/staleness sweep. No markdown linter exists, so "tests" are hook-pass + grep checks.

---

Pre-flight (read once before starting)

• rbw must be unlocked before every commit (the pre-commit ansible-lint hook decrypts vault.yml). Run rbw unlocked (exit 0 = good); if not, stop and ask the user to rbw unlock.
• Commit style: one commit per task, imperative subject ≤72 chars.
• Order matters: Task 1 (ADR-016) lands first — every later task links to it.
• Spec reference: docs/superpowers/specs/2026-06-05-mesh-vpn-netbird-design.md.
• Branch: start by creating chore/mesh-vpn-netbird-docs off main (the controller does this before dispatching Task 1; do not implement on main).

---

File map

File	Action	Responsibility after change
docs/decisions/016-mesh-vpn.md	Create	Home of record for the NetBird mesh decision
docs/decisions/007-network.md	Modify	VLAN-99 WireGuard retired; askari rides the mesh + hosts the coordinator
docs/decisions/015-control-host.md	Modify	Resolve deferred item #1 (mesh = NetBird on askari)
docs/security/accepted-risks.md	Modify	Replace R3 placeholder with the concrete residual risk
docs/CAPABILITIES.md	Modify	VPN row decided: NetBird, self-hosted
STATUS.md	Modify	Two rows: NetBird coordinator + agent enrollment (designed, not built)
CLAUDE.md	Modify	ADR-016 in Further reading

---

Task 1: Author ADR-016 (the home of record)

Files:

• Create: docs/decisions/016-mesh-vpn.md

• Step 1: Create the ADR file

Create docs/decisions/016-mesh-vpn.md with exactly this content (preserve em-dashes —, backticks, table pipes, and the verified: stamps):

# ADR-016 — Mesh VPN (NetBird, self-hosted on `askari`)

## Context

`ubongo` (ADR-015) needs remote SSH access from anywhere without exposing anything to
the public internet; ADR-015 deferred the mechanism. ADR-007 already commits to
WireGuard-via-OPNsense for the `vpn` VLAN (VLAN 99, `10.99.0.0/24`: `askari` + road
warriors), and `docs/CAPABILITIES.md` flagged NetBird (mesh) as a real alternative to
weigh. This ADR settles it.

## Decision

A single **NetBird** mesh is the sole remote-access overlay, self-hosted on `askari`,
**replacing** ADR-007's VLAN-99 OPNsense WireGuard.

The decision in four parts:

1. **Scope — mesh replaces WireGuard.** One overlay for `ubongo`, `askari`, and
   road-warrior clients. ADR-007's VLAN-99 WireGuard design is retired.
2. **Control plane — self-hosted on `askari`.** Sovereignty (boma self-hosts
   Vaultwarden, Forgejo, DNS), no third-party trust, and an off-site coordinator that
   survives a homelab outage and stays out of the cluster it administers.
3. **Tool — NetBird.** Self-hosting selects NetBird (first-class, fully open-source
   self-host). Tailscale would mean Headscale (third-party reimplementation, partial
   parity) — ruled out below.
4. **Routing — agent on every Linux host**, not a subnet router. At boma's scale (2–5
   hosts) the "agent everywhere" cost is trivial and the `base` role already runs
   everywhere, so enrollment is one uniform task. Avoids a routing SPOF and gives
   granular per-peer ACLs. OPNsense (FreeBSD) is the one non-agent exception
   (`mgmt`/gateway reached by a single advertised route or LAN-side admin).
5. **Identity — embedded local users** (Dex in the management container); external SSO
   (Zitadel/Keycloak) stays an optional future.

## Verified facts (ADR-014)

verified: NetBird self-hosting · NetBird docs · docs.netbird.io/selfhosted · 2026-06-05
— components management+signal+dashboard+relay/TURN(Coturn), **single container since
v0.65**; **built-in local users / embedded IdP since v0.62** (external OIDC optional);
ports TCP 80/443 + UDP 3478 behind a reverse proxy; lightweight Linux + Docker Compose host.

verified: NetBird licensing · GitHub netbirdio/netbird · 2026-06-05 — AGPLv3 for
`management/`/`signal/`/`relay/`, BSD-3-Clause elsewhere; fully open source, no
open-core feature gating.

## Architecture

Data plane: peer-to-peer WireGuard. Control plane: NetBird, self-hosted on `askari`.
NetBird manages its own overlay addressing (default `100.64.0.0/10`); no boma VLAN is
allocated for it.

- `askari` (Hetzner, off-site, always-up) — runs the NetBird stack **and** is a peer.
- `ubongo` — agent.
- All Linux managed hosts — agent via the `base` role.
- Road-warrior clients (`mamba`, phone, work PC) — agent/app.
- OPNsense / `mgmt` — single non-agent exception.

## Security

- **ACLs mirror ADR-007 intent** (NetBird default-deny): mesh peers → `srv` metrics
  ports only; admin peers (`ubongo`, `mamba`) → `srv` + `mgmt`; clients → least
  privilege.
- **Enrollment via setup keys** stored in `vault.yml` (`vault.netbird.setup_key`),
  consumed by `base`; prefer ephemeral/scoped keys.
- **Host firewall:** NetBird's `wt0` interface; `base` nftables allows inbound SSH
  **only on `wt0`** (the ADR-015 pattern, fleet-wide).
- **New public surface on `askari`:** management API + dashboard (80/443) + Coturn
  (3478). Mitigated by TLS + embedded-IdP login, source-IP limits where practical,
  `base` hardening, and version-pinned NetBird (ADR-011) patched on boma's cadence.
  Recorded as accepted-risk R3.

## Recovery & operations

- **Ansible stays off the mesh:** `ubongo` reaches the fleet by LAN IP (ADR-009); a
  mesh/coordinator outage never blocks on-LAN runs.
- **Bootstrap order:** stand up the coordinator on `askari` → enroll `ubongo` →
  `base` enrolls the fleet.
- **Coordinator survival:** off-site on `askari` ⇒ mesh survives a homelab outage.
  NetBird's management datastore is backed up encrypted off `askari` (synced to
  `ubongo`/`mamba`); peers keep last-known config through a brief coordinator outage.
- **`askari` is Ansible-managed:** its own inventory group, `base` role, plus a
  dedicated `netbird_coordinator` service role (one service = one role, ADR-004; with
  `SECURITY.md`). Agent install/enrollment lives in `base`. NetBird server + agents are
  version-pinned (ADR-011). boma's `dns` role stays authoritative for
  `boma.baobab.band`; NetBird built-in DNS scoped/off.

## Status

Designed, not built — depends on the unbuilt `base` role and service-role machinery
(STATUS.md). This ADR records the decision and doc reconciliation; role tasks land when
`base` exists.

## What was ruled out

| Option | Reason |
|---|---|
| Plain OPNsense WireGuard (ADR-007 as-is) | No identity/ACL layer, manual peer config; the operator wants policy-based mesh access and easy multi-device enrollment. |
| Tailscale (hosted coordinator) | Third-party trust for the control plane; against boma's self-hosting ethos. Its recovery benefit is matched by a self-hosted coordinator off-site on `askari`. |
| Tailscale + Headscale | Headscale is a third-party reimplementation with partial parity and no vendor support — weaker than NetBird's first-class self-hosting. |
| Coordinator on the cluster | Recreates the chicken-and-egg ADR-015 escapes and dies with the homelab. `askari` instead. |
| Subnet router via `ubongo` | Makes `ubongo` a routing SPOF; `askari` goes blind to `srv` when `ubongo` is down. Agent-per-host instead. |
| Standalone IdP (Zitadel/Keycloak) now | Heavy for one operator; embedded local users suffice. |

See also: ADR-007 (network — amended), ADR-015 (control host), ADR-002 (security),
ADR-011 (version pinning), ADR-004 (one service = one role), ADR-009 (TF↔Ansible
handoff), ADR-013 (heritage — V4 ran WireGuard; NetBird is translated, not transplanted).

• Step 2: Verify and commit

Run: rbw unlocked && pre-commit run --files docs/decisions/016-mesh-vpn.md Expected: Passed/Skipped (ansible-lint Skipped for non-YAML).

git add docs/decisions/016-mesh-vpn.md
git commit -m "Add ADR-016 (mesh VPN — NetBird self-hosted on askari)"

---

Task 2: Amend ADR-007 (retire VLAN-99 WireGuard, askari on the mesh)

Files:

• Modify: docs/decisions/007-network.md

Read the file first, then make FOUR exact edits. Preserve em-dashes —, backticks, table pipes.

• Step 1: Update the VLAN-99 row in the VLAN design table

Find:

| 99 | `vpn` | `10.99.0.0/24` | WireGuard peers. `askari` (Hetzner) + road-warrior clients. |

Replace with:

| 99 | `vpn` | _(retired)_ | **Replaced by the NetBird mesh (ADR-016).** Remote access for `ubongo`, `askari`, and road-warrior clients rides a self-hosted NetBird overlay, not an OPNsense WireGuard subnet. `10.99.0.0/24` is freed. |

• Step 2: Replace the VLAN-99 addressing subsection

Find:

### VLAN 99 — vpn (10.99.0.0/24) — WireGuard

| Address | Host |
|---|---|
| `10.99.0.1` | OPNsense (WireGuard endpoint) |
| `10.99.0.2` | `askari` (Hetzner VPS) |
| `10.99.0.10`+ | Road-warrior clients |

Replace with:

### VLAN 99 — vpn — retired

The OPNsense WireGuard VPN (`10.99.0.0/24`) is **replaced by the NetBird mesh**
(ADR-016). Remote access for `ubongo`, `askari`, and road-warrior clients rides a
self-hosted NetBird overlay — data plane peer-to-peer WireGuard, control plane
NetBird self-hosted on `askari`. NetBird manages its own overlay addressing
(default `100.64.0.0/10`); no boma VLAN/subnet is allocated for it, and
`10.99.0.0/24` is freed.

• Step 3: Update the two vpn rows in the OPNsense firewall-rules table

Find:

| `vpn` | `srv` (metrics ports) | allow (monitoring) |
| `vpn` | `mgmt` | allow (administration from askari) |

Replace with:

| mesh peers | `srv` (metrics ports) | allow (monitoring) — enforced by NetBird ACLs, not OPNsense (ADR-016) |
| mesh peers | `mgmt` | allow (administration) — enforced by NetBird ACLs (ADR-016) |

• Step 4: Rewrite the "External monitoring — askari" section

Find:

`askari` (Hetzner VPS) connects via WireGuard to OPNsense (`10.99.0.1`).
Its peer address is `10.99.0.2`. OPNsense routes `10.99.0.0/24` into the VPN
tunnel and allows `askari` narrow access to `srv` metrics endpoints and `mgmt`
for administration.

`askari` is provisioned and managed independently of the Proxmox cluster — it
must be reachable even when the homelab is down (its entire purpose).
FQDN: `askari.baobab.band`.

Replace with:

`askari` (Hetzner VPS) is a peer on the **NetBird mesh** (ADR-016) and also **hosts
the self-hosted NetBird coordinator** (management/signal/relay). It reaches `srv`
metrics endpoints and `mgmt` for administration over the mesh, scoped by NetBird
ACLs — no OPNsense WireGuard tunnel and no `10.99.0.0/24` routing.

`askari` is provisioned and managed independently of the Proxmox cluster — it must
be reachable even when the homelab is down (its entire purpose), which is also why
the mesh coordinator lives here: an off-site control plane survives a homelab outage.
FQDN: `askari.baobab.band`.

• Step 5: Verify and commit

Run: rbw unlocked && pre-commit run --files docs/decisions/007-network.md Expected: Passed/Skipped.

git add docs/decisions/007-network.md
git commit -m "ADR-007: retire VLAN-99 WireGuard for the NetBird mesh (ADR-016)"

---

Task 3: Resolve ADR-015 deferred item #1

Files:

• Modify: docs/decisions/015-control-host.md

Read the file first, then make THREE exact edits.

• Step 1: Update provisioning step 3

Find:

3. Join the mesh VPN (choice deferred — see below).

Replace with:

3. Join the mesh VPN — NetBird, self-hosted on `askari` (ADR-016).

• Step 2: Update the Access & security mesh line

Find:

- Remote access is via the **mesh VPN** (choice deferred). SSH to `ubongo` over the
  mesh; nothing is published to the public internet — this stays inside ADR-002.

Replace with:

- Remote access is via the **mesh VPN** — NetBird, self-hosted on `askari` (ADR-016).
  SSH to `ubongo` over the mesh; nothing is published to the public internet — this
  stays inside ADR-002.

• Step 3: Resolve deferred item #1

Find:

1. **Mesh VPN choice** — Tailscale vs NetBird, hosted vs self-hosted. Recovery
   dimension: a hosted coordinator keeps the mesh up when the cluster is down; a
   self-hosted coordinator must live off-cluster (on `ubongo`), never on the fleet,
   or it recreates the chicken-and-egg.

Replace with:

1. **Mesh VPN choice — RESOLVED (ADR-016):** NetBird, self-hosted on `askari`
   (off-site, so it survives a homelab outage and stays out of the cluster it
   administers). Replaces ADR-007's OPNsense WireGuard.

• Step 4: Verify and commit

Run: rbw unlocked && pre-commit run --files docs/decisions/015-control-host.md Expected: Passed/Skipped.

git add docs/decisions/015-control-host.md
git commit -m "ADR-015: resolve mesh-VPN deferral — NetBird on askari (ADR-016)"

---

Task 4: Replace accepted-risks R3 with the concrete residual risk

Files:

• Modify: docs/security/accepted-risks.md

Read the file first, then make ONE exact edit. (The row is long — match it whole.)

• Step 1: Replace the R3 row

Find:

| R3 | **Mesh-VPN coordinator dependency (pending VPN choice)** — remote SSH to the control node `ubongo` (ADR-015) rides a mesh VPN whose coordination plane may be a third party (e.g. hosted Tailscale/NetBird) | A hosted coordinator keeps the mesh up when the cluster is down, which *helps* recovery; nothing is exposed to the public internet (ADR-002 preserved). Provisional — finalised when the VPN is chosen (separate discussion) | The VPN choice is settled (replace this entry with the concrete decision); a self-hosted coordinator is adopted; the provider's trust/security posture changes |

Replace with:

| R3 | **Self-hosted mesh control plane is a public target on `askari`** — the NetBird coordinator (ADR-016) exposes a management API + dashboard (TCP 80/443) and Coturn (UDP 3478) on `askari`'s public IP; the management API controls the whole mesh | Self-hosting means **no third-party trust** and an off-site control plane that survives a homelab outage (boma's sovereignty ethos). Residual surface is on `askari` (already a public VPS) and is mitigated: TLS + embedded-IdP login, source-IP restriction where practical, `base` hardening, version-pinned NetBird (ADR-011) patched on boma's cadence | A coordinator compromise or unpatched NetBird CVE; the management plane is reachable without auth/IP-limits; the operational burden makes a hosted coordinator worth reconsidering |

• Step 2: Bump the "Last reviewed" date

Find:

_Last reviewed: 2026-06-05. The prior gaps

This already reads 2026-06-05 (today) from the previous work, so no change is needed — confirm it says 2026-06-05 and move on. (If it shows an earlier date, set it to 2026-06-05.)

• Step 3: Verify and commit

Run: rbw unlocked && pre-commit run --files docs/security/accepted-risks.md Expected: Passed/Skipped.

git add docs/security/accepted-risks.md
git commit -m "accepted-risks: R3 now the concrete NetBird coordinator risk"

---

Task 5: Update the CAPABILITIES VPN row

Files:

• Modify: docs/CAPABILITIES.md

Read the file first, then make ONE exact edit.

• Step 1: Replace the VPN / remote access row

Find:

| VPN / remote access | Netbird · *or* OPNsense WireGuard | P | candidate | Secure remote access to `srv`/`mgmt` | ⚠️ ADR-007 commits WireGuard-via-OPNsense; Netbird (mesh) is a real alternative to weigh |

Replace with:

| VPN / remote access | NetBird (self-hosted on `askari`) | P | core | Secure mesh remote access to `srv`/`mgmt` | **Decided (ADR-016):** NetBird mesh replaces ADR-007 OPNsense WireGuard |

• Step 2: Verify and commit

Run: rbw unlocked && pre-commit run --files docs/CAPABILITIES.md Expected: Passed/Skipped.

git add docs/CAPABILITIES.md
git commit -m "CAPABILITIES: VPN decided — NetBird self-hosted (ADR-016)"

---

Task 6: Add NetBird rows to STATUS.md

Files:

• Modify: STATUS.md

Read the file first, then make ONE exact edit (add two rows after the ubongo row).

• Step 1: Add the two rows

Find:

| `ubongo` — physical control / AI-worker host | ADR-015 | Replaces the cluster control VM with a dedicated always-on x86 box outside the cluster. Decision recorded; box not yet acquired/installed, not in inventory. |

Replace with that SAME line followed by the two new rows:

| `ubongo` — physical control / AI-worker host | ADR-015 | Replaces the cluster control VM with a dedicated always-on x86 box outside the cluster. Decision recorded; box not yet acquired/installed, not in inventory. |
| NetBird mesh — coordinator on `askari` | ADR-016 | Self-hosted NetBird control plane (management/signal/relay) on askari; replaces ADR-007 WireGuard. Decision recorded; not deployed (askari + service-role machinery not built). |
| NetBird agent enrollment in `base` | ADR-016 | Every Linux host joins the mesh via the base role (setup keys in vault); SSH allowed only on `wt0`. Designed; base role not built. |

• Step 2: Verify and commit

Run: rbw unlocked && pre-commit run --files STATUS.md Expected: Passed/Skipped.

git add STATUS.md
git commit -m "STATUS: record NetBird mesh (coordinator + base enrollment)"

---

Task 7: Link ADR-016 from CLAUDE.md

Files:

• Modify: CLAUDE.md

Read the file first, then make ONE exact edit.

• Step 1: Add the Further reading row after Network topology

Find:

| Network topology       | `docs/decisions/007-network.md`       |

Replace with that SAME line followed by the new row:

| Network topology       | `docs/decisions/007-network.md`       |
| Mesh VPN (NetBird, self-hosted) | `docs/decisions/016-mesh-vpn.md` |

• Step 2: Verify and commit

Run: rbw unlocked && pre-commit run --files CLAUDE.md Expected: Passed/Skipped.

git add CLAUDE.md
git commit -m "CLAUDE.md: link ADR-016 (mesh VPN)"

---

Task 8: Final consistency sweep

Files: none modified (verification only)

• Step 1: Confirm no doc still treats OPNsense WireGuard / 10.99 as the active remote-access path, and no "pending/deferred VPN" language remains

Run:

grep -rniE "choice deferred|pending VPN choice|10\.99\.0|WireGuard (endpoint|peers|to OPNsense)" docs/ CLAUDE.md STATUS.md | grep -vE "superpowers/(plans|specs)/"

Expected: the ONLY hits are in 007-network.md and 016-mesh-vpn.md, where they describe the retirement of 10.99.0.0/24 (e.g. "10.99.0.0/24 is freed", "no 10.99.0.0/24 routing") — those are correct and expected. There must be no hit that still treats OPNsense WireGuard or 10.99.0.x as the live remote-access path, and no choice deferred / pending VPN choice anywhere. Legitimate mentions of "WireGuard" as NetBird's data plane are fine and won't match this pattern (it only matches WireGuard endpoint|peers|to OPNsense). If a canonical doc still names the WireGuard VPN as live, fix it as in the relevant task above and amend that commit.

• Step 2: Confirm ADR-016 exists and is cross-linked

Run:

test -f docs/decisions/016-mesh-vpn.md && echo "ADR-016 present"
grep -rl "ADR-016\|016-mesh-vpn" docs/ CLAUDE.md STATUS.md | grep -vE "superpowers/(plans|specs)/"

Expected: the file exists and the referencing docs (007, 015, accepted-risks, CAPABILITIES, STATUS, CLAUDE.md) appear.

• Step 3: Full hook run

Run: rbw unlocked && pre-commit run --all-files Expected: all hooks Passed/Skipped. Fix anything that fails (most likely trailing whitespace / end-of-file) and amend the owning commit.

• Step 4: Push (only if the user asks)

Per CLAUDE.md, push to origin is the off-machine backup. If the user wants it pushed:

git push origin <branch-or-main-after-merge>

---

Self-review notes (author)

• Spec coverage: decision/architecture/security/recovery → Task 1 (ADR-016); the spec's "Documentation & implementation changes" table → Tasks 2–7; deferrals (external SSO, OPNsense mesh specifics, role implementation) are recorded in ADR-016/STATUS, not implemented here (correct — they need the unbuilt base/service-role machinery). ✓
• Not in scope (intentional): the netbird_coordinator service role, the base-role agent task, vault setup_key material, and any live deployment — all wait on base/service-role machinery (STATUS-honest). ✓
• No placeholders: every edit shows exact find/replace text; the _(retired)_ token in ADR-007 is deliberate table content. ✓
• Name consistency: ADR file is 016-mesh-vpn.md everywhere; vault.netbird.setup_key, netbird_coordinator, and wt0 are used identically across ADR-016 and the sweep. ✓