# Design — Control / development / AI-worker host (`ubongo`)

- **Date:** 2026-06-05
- **Status:** Approved design — pending implementation plan
- **Supersedes (in part):** the "control node is a dedicated VM on the cluster"
  assumption in ADR-001 / ADR-005 / ADR-009
- **Becomes:** ADR-015 (this design is the basis for that ADR)

---

## Problem

Today the control node — the host that runs Terraform and Ansible — is defined as a
**single Debian 13 VM on the Proxmox cluster**, manually provisioned as the one
documented exception to "Terraform owns VM existence" (ADR-009). The ADRs treat it
purely as a control-plane runner.

That framing fails four things the user actually needs, all confirmed as drivers:

1. **Cold-start bootstrap** — the VM that runs Terraform/Ansible can't exist until
   something else creates it; the manual bootstrap is awkward and circular.
2. **Always-on availability** — the user wants to SSH in from a work PC (or anywhere)
   to fire off Claude Code commands. A VM on the cluster is gone whenever the cluster
   is down or being rebuilt.
3. **Recovery / disaster** — the tool you'd use to rebuild the cluster must not live
   *inside* the thing it rebuilds.
4. **Dev ergonomics** — a comfortable, persistent home for Claude Code + the repo,
   not entangled with production VM lifecycle.

A laptop-only answer fails always-on (not always carried) and recovery. A VM-only
answer fails cold-start and recovery. A small **dedicated always-on physical machine
outside the cluster** satisfies all four.

---

## Decision

Introduce **`ubongo`**: a single dedicated x86-64 mini-PC, always-on, living
**outside** the Proxmox cluster. It becomes *the* control node and collapses four
roles into one box:

- Terraform + Ansible runner (control plane)
- Claude Code / AI-worker host the user SSHes into
- Local test runner (Molecule/Docker, lint, and later the browser stack)
- Persistent dev home for the repo

There is **no longer a control VM on the cluster.** The `control` inventory group now
points at this physical box. This *strengthens* the ADR-009 control-node exception:
it is genuinely outside Terraform's world, not a VM pretending to be the exception.
Every other host stays a Terraform-managed VM exactly as designed.

`ubongo` runs **plain Debian 13** (matches the fleet — the `base` role applies). It is
not a hypervisor.

### Name

`ubongo` (Swahili: *brain*), consistent with the fleet's Swahili theme (`boma`,
`nyumbani`, `askari`, `mamba`).

---

## Hardware target

| Spec | Target | Why |
|---|---|---|
| CPU | 4 cores, x86-64 (Intel N100-class or better) | Molecule containers + Chromium prefer x86 |
| RAM | 16 GB | Docker + headless Chromium + toolchain headroom |
| Disk | 250 GB SSD/NVMe | Docker images, molecule layers, repos, browser cache |
| Network | Wired GbE | Always-on reliability over Wi-Fi |
| Power | Low draw (≤15 W idle) | Runs 24/7 |

Indicative hardware: a refurb Dell/Lenovo/HP micro (USFF) or an N100 mini-PC,
roughly €150–250.

**Sizing rationale.** Claude Code itself is light — the model runs in Anthropic's
cloud; the box only runs the CLI, git, Ansible, Terraform, `rbw`, SSH. The real
sizing driver is **all testing being local**: Molecule (Docker — the heavy one), lint,
and later a headless-Chromium/Playwright browser stack for service-UI verification.
That combination is firmly mini-PC / small-server class, not a Raspberry Pi.

---

## Provisioning (bootstrap path)

Manual, like today's control-node exception — but on bare metal instead of a clone:

1. Install Debian 13 on the box (one-time, by hand).
2. `git clone` the repo; `make setup`; `make collections`; set up `rbw` + unlock.
3. Join the mesh VPN (choice TBD — see Deferred).
4. From then on `ubongo` manages every other host normally. Ansible manages *it* for
   baseline config via the `control` group (`base` role: SSH, firewall, updates,
   auditd) — but never the `docker_host` role. It runs no services.

---

## Access & security

- **Remote access via the mesh VPN** (choice TBD). The user SSHes to `ubongo` over the
  mesh from work PC, laptop, or phone. **Nothing is published to the public internet**
  — SSH-over-mesh keeps the design fully inside ADR-002 (no LAN/WAN exposure without
  reverse-proxy + auth).
- **Hardening.** `ubongo` runs the `base` role like every host: SSH hardening,
  nftables default-deny, fail2ban, auditd, unattended-upgrades. Inbound SSH is allowed
  **only on the mesh interface** — denied on the physical NIC. Even on the LAN it is
  not an open SSH target.
- **Third-party dependency.** A hosted mesh coordinator is a third party. This is a
  deliberate trade: a hosted control plane keeps the mesh up when the cluster is down
  (helps recovery). A self-hosted coordinator on the cluster would recreate the
  chicken-and-egg — so if self-hosted, it must live on `ubongo` or off-cluster, never
  on the fleet. To be logged in `accepted-risks.md` once the VPN is chosen.

---

## Recovery model

`ubongo` is now the rebuild tool, so three things must survive a full cluster loss:

1. **The box / its data.** `mamba` (laptop) stays a **break-glass clone**: repo +
   toolchain + mesh + `rbw`, able to drive the fleet if `ubongo` dies. Two machines
   that can drive the fleet, not one.
2. **Terraform state.** Lives on `ubongo`, backed up **encrypted off-box** (synced to
   `mamba`). For a 2–5 VM fleet it is also reconstructable via `terraform import`, so
   this is belt-and-suspenders, not load-bearing.
3. **The vault password.** `ubongo` gets the vault master password from Vaultwarden via
   `rbw`. `rbw` keeps a **local encrypted copy** of the Vaultwarden vault and decrypts
   it **offline** with the user's Vaultwarden master password — no live server needed
   for already-synced entries. So provided (a) `rbw` has synced at least once and (b)
   the user keeps their **Vaultwarden master password** offline (memorised + paper in a
   safe), `ubongo` can decrypt the Ansible vault with the whole cluster down. Mirror
   the same onto `mamba`.

**Why not mirror/replicate Vaultwarden onto `ubongo`?** It would make the control node
*run a service* (against its remit + adds attack surface), add DB-replication
complexity, and **still** require the Vaultwarden master password to read anything.
There is always exactly **one irreducible offline root secret** — make it the
Vaultwarden master password, and let `rbw`'s local cache make everything else
self-serve offline. (If full disaster access to *all* secrets — router, Proxmox UI —
is wanted, that same `rbw` cache already covers it; optionally add a scheduled
encrypted `rbw export` as extra insurance.)

> **To verify (ADR-014, security-relevant):** the "`rbw` decrypts its local cache fully
> offline" behaviour is the load-bearing assumption of the recovery model. Confirm it
> against `rbw`'s docs/version during implementation before relying on it.

---

## Testing implication

All testing runs on `ubongo`:

- **Level 1 (Molecule)** — Docker on `ubongo`.
- **Lint** — on `ubongo`.
- **Level 2 / 3** (staging deploy, external smoke) — driven from `ubongo` as before.

A future **service-UI acceptance level** (Claude driving a headless browser against a
deployed service: load the UI, create test users, exercise features, hand the user a
manual test script) is anticipated. `ubongo` is *sized* for it now (Chromium +
Playwright headroom). The harness itself is a **separate spec** (see Deferred).

---

## Documentation changes

A new **ADR-015 — Control / development / AI-worker host (`ubongo`)** is the home of
record. Other docs get small amendments that link to it:

| Doc | Change |
|---|---|
| ADR-015 (new) | Full record of this design. |
| ADR-001 (architecture) | Control node: "dedicated Debian 13 VM on the cluster" → "dedicated physical x86 machine *outside* the cluster (`ubongo`)". |
| ADR-005 (bootstrapping) | Control-node section: "clone the cloud-init template by hand" → "install Debian 13 on the physical box". |
| ADR-009 (provisioning handoff) | Strengthen the control-node exception: now genuinely physical/outside Terraform. |
| ADR-008 (testing) | "runs on the control node or in CI" → all levels run on `ubongo`; add a stub for the future service-UI acceptance level. |
| ADR-012 / `docs/hardware/reference.md` | Add `ubongo` to the node-capacity table (physical compute, though outside the cluster). |
| `docs/runbooks/new-host.md` | Update the control-node bootstrap procedure (bare-metal Debian install, not `qm clone`). |
| `docs/runbooks/rotate-secrets.md` | Add the offline vault-password break-glass requirement. |
| `docs/security/accepted-risks.md` | Reserve an entry for the mesh-VPN third-party coordinator — pending the VPN choice. |
| `STATUS.md` | Add a row: `ubongo` — *designed, not built*. |
| `CLAUDE.md` | One-line touch to the inventory/`control`-group description if needed. |

---

## Explicitly deferred (separate specs / discussions)

1. **Mesh VPN choice** — Tailscale vs NetBird, hosted vs self-hosted. Carries the
   recovery-dimension note above (hosted coordinator helps recovery; self-hosted must
   be off-cluster). Its own discussion.
2. **Browser-E2E verification harness** — Playwright/headless-Chromium driving live
   service UIs, test-user generation, screenshot-back-to-Claude, and the new ADR-008
   level. `ubongo` is sized for it now; the harness is designed later.
3. **`rbw` offline-cache verification** — a to-verify task during implementation
   (ADR-014), before relying on offline decryption.

---

## What was ruled out

| Option | Reason |
|---|---|
| Keep control node as a cluster VM | Fails cold-start and recovery (rebuild tool lives inside the thing it rebuilds); fails always-on (dies with the cluster). |
| Laptop-only (`mamba` for everything) | Fails always-on (not always carried). Retained instead as the break-glass backup. |
| Split roles (control VM + thin jump box) | Two places to maintain the toolchain, control plane split in two, heavy local testing back on a cluster VM — more moving parts, less benefit. |
| Mirror/replicate Vaultwarden onto `ubongo` | Makes the control node run a service (against its remit), adds DB-replication complexity, and still needs the Vaultwarden master password. `rbw`'s local cache achieves offline decryption without it. |
| Self-hosted mesh coordinator on the cluster | Recreates the chicken-and-egg the whole design escapes. If self-hosted, it lives off-cluster. |
| Raspberry Pi as the box | Could just run Molecule, but chokes running Docker + Chromium + toolchain together. x86 mini-PC instead. |