boma/docs/superpowers/specs/2026-06-05-mesh-vpn-netbird-design.md

Design — Mesh VPN (NetBird, self-hosted on askari)

• Date: 2026-06-05
• Status: Approved design — pending implementation plan
• Resolves: ADR-015 deferred item #1 (mesh VPN choice) and the accepted-risks.md R3 "pending VPN choice" placeholder
• Amends: ADR-007 (retires the VLAN-99 OPNsense WireGuard design)
• Becomes: ADR-016 (this design is the basis for that ADR)

---

Problem

ubongo (ADR-015) needs remote SSH access from anywhere (work PC, laptop, phone) without exposing anything to the public internet. ADR-015 left the access mechanism — the "mesh VPN" — deferred to this discussion.

Meanwhile ADR-007 already commits to WireGuard-via-OPNsense for the vpn VLAN (VLAN 99, 10.99.0.0/24): askari (the off-site Hetzner monitoring VPS) peers to OPNsense, plus road-warrior clients. And docs/CAPABILITIES.md already flags the open question: "ADR-007 commits WireGuard-via-OPNsense; Netbird (mesh) is a real alternative to weigh."

So the real decision is three-cornered (plain OPNsense WireGuard vs NetBird vs Tailscale), with an architectural sub-question of whether a mesh replaces or coexists with the ADR-007 WireGuard.

Decisions (as settled)

1. Scope — the mesh replaces WireGuard. A single overlay becomes the sole remote-access path for ubongo, askari, and road-warrior clients. ADR-007's VLAN-99 OPNsense WireGuard design is retired.
2. Control plane — self-hosted, on askari. Maximum sovereignty (boma already self-hosts Vaultwarden, Forgejo, its own DNS), no third-party trust, and an off-site coordinator that survives a homelab outage and stays out of the cluster it administers.
3. Tool — NetBird. Self-hosting on askari selects NetBird: it is designed to be self-hosted as a first-class, fully open-source stack. (Tailscale's self-host path means Headscale, a separate third-party reimplementation with partial parity — ruled out below.)
4. Routing — NetBird agent on every (Linux) host, not a subnet router. At boma's scale (2–5 hosts, treated as individuals) the usual "agent everywhere" downside is moot, and the base role already runs on every host, so enrollment is one uniform role task. Avoids a routing single-point-of-failure and gives granular per-peer ACLs that match ADR-007's firewall intent. One exception: OPNsense (FreeBSD) is not a first-class NetBird agent target, so mgmt/gateway reachability is handled by a single advertised route or by administering OPNsense from an on-LAN meshed peer.
5. Identity — embedded local users (Dex, built into the management container), not a standalone Zitadel/Keycloak. YAGNI for a single operator; external SSO remains a documented future option.

Verified facts (ADR-014)

verified: NetBird self-hosting architecture · NetBird docs · docs.netbird.io/selfhosted · 2026-06-05

• Components: management + signal + dashboard + relay/TURN (Coturn). Since v0.65 the core services are merged into a single container; deploy via Docker Compose.
• Identity: since v0.62, built-in local users with an embedded IdP (Dex); external OIDC IdPs (Zitadel, Keycloak, Authentik, Okta, …) are optional, not required.
• Ports (behind reverse proxy): TCP 80/443 + UDP 3478 (STUN/TURN).
• Host: a Linux VM + Docker Compose + a domain name; lightweight.

verified: NetBird licensing · GitHub netbirdio/netbird · 2026-06-05

• Dual license: AGPLv3 for management/, signal/, relay/; BSD-3-Clause elsewhere. Fully open source, self-hostable, no open-core feature gating.

---

Architecture & topology

A single NetBird mesh is the sole remote-access overlay, replacing ADR-007's VLAN-99 WireGuard. Data plane is peer-to-peer WireGuard; control plane is self-hosted NetBird on askari.

askari's dual role. askari (Hetzner, off-site, always-up, independent of the cluster per ADR-007) runs the NetBird management stack (single container: management + signal + dashboard + Coturn, behind a reverse proxy on TCP 80/443 + UDP 3478) and is itself a mesh peer. Off-site hosting is what makes the mesh survive a full homelab outage and keeps the coordinator out of the cluster it administers (no chicken-and-egg).

Peers:

• askari — coordinator + peer.
• ubongo (control/AI-worker host) — agent.
• All Linux managed hosts (dns1/2, proxy, …) — agent via the base role.
• Road-warrior clients — mamba, phone, work PC — agent/app.
• OPNsense / mgmt — the single non-agent exception (advertised route or LAN-side admin from a meshed peer).

Retired: ADR-007's VLAN-99 WireGuard endpoint on OPNsense and the 10.99.0.0/24 peer scheme. askari reaches srv/mgmt over the mesh under NetBird ACLs instead of OPNsense routing 10.99.0.0/24.

---

Security model, ACLs, and attack surface

ACL policy mirrors ADR-007's firewall intent (NetBird is default-deny):

• vpn peers → srv metrics ports only (askari's monitoring scope).
• admin peers (ubongo, mamba) → srv + mgmt for administration.
• road-warrior clients → only what each needs; nothing by default.

Enrollment via setup keys. Hosts join non-interactively using NetBird setup keys, stored in vault.yml as vault.netbird.setup_key and consumed by the base role. Prefer ephemeral/scoped keys (ADR-002).

Host firewall interaction. NetBird creates a wt0 mesh interface. The base role's nftables default-deny allows inbound admin (SSH) only on wt0, denied on the physical NIC — the pattern ADR-015 set for ubongo, now applied fleet-wide. Mesh

• nftables are defence-in-depth.

The new attack surface — a public control plane on askari. Today askari exposes a WireGuard UDP port; with NetBird self-hosted it exposes the **management API

• dashboard (80/443)** and Coturn (3478) publicly, and the management API is keys-to-the-kingdom for the whole mesh. Mitigations baked in:

• Dashboard/API behind TLS + the embedded IdP login; source-IP restrictions where practical.
• askari runs base hardening (already a public managed host) and NetBird is version-pinned (ADR-011) and patched on boma's cadence — self-hosting means owning the CVE cadence (AGPLv3 server).

Net vs ADR-002: nothing from the cluster is publicly exposed; the only public surface is on askari (a public VPS by design), shifting from "WireGuard port" to "NetBird control plane."

---

Recovery, bootstrap ordering, and operations

Ansible's control path stays off the mesh. ubongo is on the LAN and reaches the fleet by LAN IP (ADR-009). The mesh only provides external reach to ubongo/the fleet, so a mesh/coordinator outage never blocks on-LAN Ansible runs and there is no chicken-and-egg in the critical path.

Bootstrap order (askari-first):

1. Stand up the NetBird coordinator on askari.
2. Enroll ubongo.
3. base role enrolls the rest of the fleet via setup keys from vault.

Recovery. Coordinator off-site on askari ⇒ the mesh survives a full homelab outage. Two must-haves:

• Back up NetBird's management datastore off askari — encrypted, synced to ubongo/mamba. If askari dies, restore the coordinator; peers re-enroll.
• Existing peer tunnels keep running on last-known config through a brief coordinator outage; only changes/new enrollments need it live — so askari is important but not instantly fatal.

askari becomes Ansible-managed. It joins the inventory under its own group and gets the base role plus a dedicated netbird_coordinator service role (one service = one role per ADR-004, with its own SECURITY.md per the service-role standard). Agent install/enrollment lives in base.

DNS & versions. boma's dns role stays authoritative for boma.baobab.band; NetBird's built-in DNS is scoped/off to avoid overlap. NetBird server (on askari) and agents (via base) are version-pinned (ADR-011).

---

Documentation & implementation changes

This is a substantial decision → its own ADR, with amendments linking to it.

Doc	Change
ADR-016 (new)	Home of record for this design.
ADR-007 (network)	Replace the VLAN-99 WireGuard section + 10.99.0.0/24 scheme with the NetBird mesh; update the firewall-intent table and the askari external-monitoring section to ride the mesh.
ADR-015 (control host)	Resolve deferred item #1: mesh VPN = NetBird self-hosted on askari; update the access/recovery notes.
docs/security/accepted-risks.md	Replace R3 ("pending VPN choice") with the concrete residual risk: self-hosted coordinator = no third-party trust, but a public NetBird control plane on askari to harden + patch.
docs/CAPABILITIES.md	Resolve the VPN row (line ~29): decided — NetBird mesh, self-hosted on askari.
STATUS.md	Add rows (designed, not built): NetBird coordinator on askari; NetBird agent enrollment in base.
base role (when built)	Install + enroll the NetBird agent; nftables allows SSH only on wt0.
netbird_coordinator service role (new, when built)	Deploys the NetBird stack on askari; populated SECURITY.md; molecule scenario.
requirements.yml	Only if a task needs a new collection module (ADR dependencies policy).

Scope note: like the ubongo work, most implementation here waits on the base and service-role machinery that STATUS.md lists as not-yet-built. This spec settles the decision and the doc reconciliation; the role tasks land when base is built.

---

Deferred / out of scope

1. External SSO IdP (Zitadel/Keycloak) — embedded local users now; SSO later if a second operator or service-SSO need appears.
2. OPNsense mesh integration specifics — the exact mgmt reachability mechanism (single advertised route vs LAN-side admin) is settled during implementation when OPNsense automation is built.
3. The base / netbird_coordinator role implementation — depends on the unbuilt base role and service-role standard.

---

What was ruled out

Option	Reason
Plain OPNsense WireGuard (ADR-007 as-is)	No identity/ACL layer, manual peer config, OPNsense-centric; the operator wants a mesh with policy-based access and easy multi-device enrollment.
Tailscale (hosted coordinator)	Adds a third-party trust dependency for the control plane; against boma's self-hosting ethos. (Hosted coordinator's recovery benefit is matched by putting a self-hosted coordinator off-site on askari.)
Tailscale + Headscale (self-hosted)	Headscale is a third-party reimplementation of Tailscale's control server with partial feature parity and no official vendor support — weaker than NetBird's first-class self-hosting.
Mesh coordinator on the cluster	Recreates the chicken-and-egg ADR-015 escapes, and dies with the homelab. askari (off-site) instead.
Subnet router via ubongo	Makes ubongo a routing SPOF; askari would go blind to srv when ubongo is down even if services are healthy. Agent-per-host instead.
Standalone IdP (Zitadel/Keycloak) now	Heavy for a single operator; embedded local users (Dex) suffice. External SSO stays a future option.

See also: ADR-007 (network), ADR-015 (control host), ADR-002 (security), ADR-011 (version pinning), ADR-004 (one service = one role), ADR-009 (TF↔Ansible handoff), ADR-013 (heritage — V4 used WireGuard; NetBird is translated, not transplanted).