110 lines
6.2 KiB
Markdown
110 lines
6.2 KiB
Markdown
# ADR-016 — Mesh VPN (NetBird, self-hosted on `askari`)
|
||
|
||
## Context
|
||
|
||
`ubongo` (ADR-015) needs remote SSH access from anywhere without exposing anything to
|
||
the public internet; ADR-015 deferred the mechanism. ADR-007 already commits to
|
||
WireGuard-via-OPNsense for the `vpn` VLAN (VLAN 99, `10.99.0.0/24`: `askari` + road
|
||
warriors), and `docs/CAPABILITIES.md` flagged NetBird (mesh) as a real alternative to
|
||
weigh. This ADR settles it.
|
||
|
||
## Decision
|
||
|
||
A single **NetBird** mesh is the sole remote-access overlay, self-hosted on `askari`,
|
||
**replacing** ADR-007's VLAN-99 OPNsense WireGuard.
|
||
|
||
The decision in four parts:
|
||
|
||
1. **Scope — mesh replaces WireGuard.** One overlay for `ubongo`, `askari`, and
|
||
road-warrior clients. ADR-007's VLAN-99 WireGuard design is retired.
|
||
2. **Control plane — self-hosted on `askari`.** Sovereignty (boma self-hosts
|
||
Vaultwarden, Forgejo, DNS), no third-party trust, and an off-site coordinator that
|
||
survives a homelab outage and stays out of the cluster it administers.
|
||
3. **Tool — NetBird.** Self-hosting selects NetBird (first-class, fully open-source
|
||
self-host). Tailscale would mean Headscale (third-party reimplementation, partial
|
||
parity) — ruled out below.
|
||
4. **Routing — agent on every Linux host**, not a subnet router. At boma's scale (2–5
|
||
hosts) the "agent everywhere" cost is trivial and the `base` role already runs
|
||
everywhere, so enrollment is one uniform task. Avoids a routing SPOF and gives
|
||
granular per-peer ACLs. OPNsense (FreeBSD) is the one non-agent exception
|
||
(`mgmt`/gateway reached by a single advertised route or LAN-side admin).
|
||
5. **Identity — embedded local users** (Dex in the management container); external SSO
|
||
(Zitadel/Keycloak) stays an optional future.
|
||
|
||
## Verified facts (ADR-014)
|
||
|
||
verified: NetBird self-hosting · NetBird docs · docs.netbird.io/selfhosted · 2026-06-05
|
||
— components management+signal+dashboard+relay/TURN(Coturn), **single container since
|
||
v0.65**; **built-in local users / embedded IdP since v0.62** (external OIDC optional);
|
||
ports TCP 80/443 + UDP 3478 behind a reverse proxy; lightweight Linux + Docker Compose host.
|
||
|
||
verified: NetBird licensing · GitHub netbirdio/netbird · 2026-06-05 — AGPLv3 for
|
||
`management/`/`signal/`/`relay/`, BSD-3-Clause elsewhere; fully open source, no
|
||
open-core feature gating.
|
||
|
||
## Architecture
|
||
|
||
Data plane: peer-to-peer WireGuard. Control plane: NetBird, self-hosted on `askari`.
|
||
NetBird manages its own overlay addressing (default `100.64.0.0/10`); no boma VLAN is
|
||
allocated for it.
|
||
|
||
- `askari` (Hetzner, off-site, always-up) — runs the NetBird stack **and** is a peer.
|
||
- `ubongo` — agent.
|
||
- All Linux managed hosts — agent via the `base` role.
|
||
- Road-warrior clients (`mamba`, phone, work PC) — agent/app.
|
||
- OPNsense / `mgmt` — single non-agent exception.
|
||
|
||
## Security
|
||
|
||
- **ACLs mirror ADR-007 intent** (NetBird default-deny): mesh peers → `srv` metrics
|
||
ports only; admin peers (`ubongo`, `mamba`) → `srv` + `mgmt`; clients → least
|
||
privilege.
|
||
- **Enrollment via setup keys** stored in `vault.yml` (`vault.netbird.setup_key`),
|
||
consumed by `base`; prefer ephemeral/scoped keys.
|
||
- **Host firewall:** `base` nftables allows inbound SSH on NetBird's `wt0` interface
|
||
(primary, WireGuard-authenticated) **and** from `ubongo`'s LAN address (secondary,
|
||
mesh-independent — required by the LAN-IP recovery path below, so a mesh/coordinator
|
||
outage never blocks on-LAN SSH). All other LAN hosts remain default-denied. This makes
|
||
explicit the control-node SSH allow that the recovery model already implied; the access
|
||
doctrine and the three-tier access ladder live in **ADR-021**.
|
||
- **New public surface on `askari`:** management API + dashboard (80/443) + Coturn
|
||
(3478). Mitigated by TLS + embedded-IdP login, source-IP limits where practical,
|
||
`base` hardening, and version-pinned NetBird (ADR-011) patched on boma's cadence.
|
||
Recorded as accepted-risk R3.
|
||
|
||
## Recovery & operations
|
||
|
||
- **Ansible stays off the mesh:** `ubongo` reaches the fleet by LAN IP (ADR-009); a
|
||
mesh/coordinator outage never blocks on-LAN runs.
|
||
- **Bootstrap order:** stand up the coordinator on `askari` → enroll `ubongo` →
|
||
`base` enrolls the fleet.
|
||
- **Coordinator survival:** off-site on `askari` ⇒ mesh survives a homelab outage.
|
||
NetBird's management datastore is backed up encrypted off `askari` (synced to
|
||
`ubongo`/`mamba`); peers keep last-known config through a brief coordinator outage.
|
||
- **`askari` is Ansible-managed:** its own inventory group `offsite_hosts` (added
|
||
manually like the control node — it is not Terraform-managed), `base` role, plus a
|
||
dedicated `netbird_coordinator` service role (one service = one role, ADR-004; with
|
||
`SECURITY.md`). Agent install/enrollment lives in `base`. NetBird server + agents are
|
||
version-pinned (ADR-011). boma's `dns` role stays authoritative for
|
||
`boma.baobab.band`; NetBird built-in DNS scoped/off.
|
||
|
||
## Status
|
||
|
||
Designed, not built — depends on the unbuilt `base` role and service-role machinery
|
||
(STATUS.md). This ADR records the decision and doc reconciliation; role tasks land when
|
||
`base` exists.
|
||
|
||
## What was ruled out
|
||
|
||
| Option | Reason |
|
||
|---|---|
|
||
| Plain OPNsense WireGuard (ADR-007 as-is) | No identity/ACL layer, manual peer config; the operator wants policy-based mesh access and easy multi-device enrollment. |
|
||
| Tailscale (hosted coordinator) | Third-party trust for the control plane; against boma's self-hosting ethos. Its recovery benefit is matched by a self-hosted coordinator off-site on `askari`. |
|
||
| Tailscale + Headscale | Headscale is a third-party reimplementation with partial parity and no vendor support — weaker than NetBird's first-class self-hosting. |
|
||
| Coordinator on the cluster | Recreates the chicken-and-egg ADR-015 escapes and dies with the homelab. `askari` instead. |
|
||
| Subnet router via `ubongo` | Makes `ubongo` a routing SPOF; `askari` goes blind to `srv` when `ubongo` is down. Agent-per-host instead. |
|
||
| Standalone IdP (Zitadel/Keycloak) now | Heavy for one operator; embedded local users suffice. |
|
||
|
||
See also: ADR-007 (network — amended), ADR-015 (control host), ADR-002 (security),
|
||
ADR-011 (version pinning), ADR-004 (one service = one role), ADR-009 (TF↔Ansible
|
||
handoff), ADR-013 (heritage — V4 ran WireGuard; NetBird is translated, not transplanted).
|