sjat/boma
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity
boma/docs/decisions/016-mesh-vpn.md

5.8 KiB
Raw Blame History

ADR-016 — Mesh VPN (NetBird, self-hosted on askari)

Context

ubongo (ADR-015) needs remote SSH access from anywhere without exposing anything to the public internet; ADR-015 deferred the mechanism. ADR-007 already commits to WireGuard-via-OPNsense for the vpn VLAN (VLAN 99, 10.99.0.0/24: askari + road warriors), and docs/CAPABILITIES.md flagged NetBird (mesh) as a real alternative to weigh. This ADR settles it.

Decision

A single NetBird mesh is the sole remote-access overlay, self-hosted on askari, replacing ADR-007's VLAN-99 OPNsense WireGuard.

The decision in four parts:

  1. Scope — mesh replaces WireGuard. One overlay for ubongo, askari, and road-warrior clients. ADR-007's VLAN-99 WireGuard design is retired.
  2. Control plane — self-hosted on askari. Sovereignty (boma self-hosts Vaultwarden, Forgejo, DNS), no third-party trust, and an off-site coordinator that survives a homelab outage and stays out of the cluster it administers.
  3. Tool — NetBird. Self-hosting selects NetBird (first-class, fully open-source self-host). Tailscale would mean Headscale (third-party reimplementation, partial parity) — ruled out below.
  4. Routing — agent on every Linux host, not a subnet router. At boma's scale (2–5 hosts) the "agent everywhere" cost is trivial and the base role already runs everywhere, so enrollment is one uniform task. Avoids a routing SPOF and gives granular per-peer ACLs. OPNsense (FreeBSD) is the one non-agent exception (mgmt/gateway reached by a single advertised route or LAN-side admin).
  5. Identity — embedded local users (Dex in the management container); external SSO (Zitadel/Keycloak) stays an optional future.

Verified facts (ADR-014)

verified: NetBird self-hosting · NetBird docs · docs.netbird.io/selfhosted · 2026-06-05 — components management+signal+dashboard+relay/TURN(Coturn), single container since v0.65; built-in local users / embedded IdP since v0.62 (external OIDC optional); ports TCP 80/443 + UDP 3478 behind a reverse proxy; lightweight Linux + Docker Compose host.

verified: NetBird licensing · GitHub netbirdio/netbird · 2026-06-05 — AGPLv3 for management//signal//relay/, BSD-3-Clause elsewhere; fully open source, no open-core feature gating.

Architecture

Data plane: peer-to-peer WireGuard. Control plane: NetBird, self-hosted on askari. NetBird manages its own overlay addressing (default 100.64.0.0/10); no boma VLAN is allocated for it.

  • askari (Hetzner, off-site, always-up) — runs the NetBird stack and is a peer.
  • ubongo — agent.
  • All Linux managed hosts — agent via the base role.
  • Road-warrior clients (mamba, phone, work PC) — agent/app.
  • OPNsense / mgmt — single non-agent exception.

Security

  • ACLs mirror ADR-007 intent (NetBird default-deny): mesh peers → srv metrics ports only; admin peers (ubongo, mamba) → srv + mgmt; clients → least privilege.
  • Enrollment via setup keys stored in vault.yml (vault.netbird.setup_key), consumed by base; prefer ephemeral/scoped keys.
  • Host firewall: NetBird's wt0 interface; base nftables allows inbound SSH only on wt0 (the ADR-015 pattern, fleet-wide).
  • New public surface on askari: management API + dashboard (80/443) + Coturn (3478). Mitigated by TLS + embedded-IdP login, source-IP limits where practical, base hardening, and version-pinned NetBird (ADR-011) patched on boma's cadence. Recorded as accepted-risk R3.

Recovery & operations

  • Ansible stays off the mesh: ubongo reaches the fleet by LAN IP (ADR-009); a mesh/coordinator outage never blocks on-LAN runs.
  • Bootstrap order: stand up the coordinator on askari → enroll ubongo → base enrolls the fleet.
  • Coordinator survival: off-site on askari ⇒ mesh survives a homelab outage. NetBird's management datastore is backed up encrypted off askari (synced to ubongo/mamba); peers keep last-known config through a brief coordinator outage.
  • askari is Ansible-managed: its own inventory group, base role, plus a dedicated netbird_coordinator service role (one service = one role, ADR-004; with SECURITY.md). Agent install/enrollment lives in base. NetBird server + agents are version-pinned (ADR-011). boma's dns role stays authoritative for boma.baobab.band; NetBird built-in DNS scoped/off.

Status

Designed, not built — depends on the unbuilt base role and service-role machinery (STATUS.md). This ADR records the decision and doc reconciliation; role tasks land when base exists.

What was ruled out

Option Reason
Plain OPNsense WireGuard (ADR-007 as-is) No identity/ACL layer, manual peer config; the operator wants policy-based mesh access and easy multi-device enrollment.
Tailscale (hosted coordinator) Third-party trust for the control plane; against boma's self-hosting ethos. Its recovery benefit is matched by a self-hosted coordinator off-site on askari.
Tailscale + Headscale Headscale is a third-party reimplementation with partial parity and no vendor support — weaker than NetBird's first-class self-hosting.
Coordinator on the cluster Recreates the chicken-and-egg ADR-015 escapes and dies with the homelab. askari instead.
Subnet router via ubongo Makes ubongo a routing SPOF; askari goes blind to srv when ubongo is down. Agent-per-host instead.
Standalone IdP (Zitadel/Keycloak) now Heavy for one operator; embedded local users suffice.

See also: ADR-007 (network — amended), ADR-015 (control host), ADR-002 (security), ADR-011 (version pinning), ADR-004 (one service = one role), ADR-009 (TF↔Ansible handoff), ADR-013 (heritage — V4 ran WireGuard; NetBird is translated, not transplanted).