diff --git a/docs/superpowers/specs/2026-06-10-backup-strategy-design.md b/docs/superpowers/specs/2026-06-10-backup-strategy-design.md new file mode 100644 index 0000000..ce62454 --- /dev/null +++ b/docs/superpowers/specs/2026-06-10-backup-strategy-design.md @@ -0,0 +1,310 @@ +# Design — Backup & disaster recovery strategy + +- **Date:** 2026-06-10 +- **Status:** Approved design — pending implementation plan +- **Resolves:** `docs/TODO.md` item 3.8 ("ensure the right things are backed up, + incl. DB dumps") and `docs/CAPABILITIES.md` §9 (backup engine / off-site / air-gap, + all "planned") +- **Grounds:** the backup substrate that ADR-011 (update management) already leans on + ("snapshot-before + backups remain the rollback mechanism", "always dumps the DB / + takes a backup first") but never defined +- **Reuses:** ADR-004 (one service = one role; per-service doc conventions), + ADR-008/017 (`VERIFY.md` per-service checks), ADR-021 (`ACCESS.md` rendered from + role `access__*` data — the same render-from-data pattern), ADR-015 (`ubongo` + recovery model; `mamba` break-glass clone) +- **Becomes:** ADR-022 (this design is the basis for that ADR) + +--- + +## Problem + +boma has no defined backup policy. The ADRs assume one exists — ADR-011 makes +"backup-first" the rule for stateful upgrades and "snapshot + backup" the rollback +path — but nothing specifies *what* gets backed up, *how* it stays consistent, *where* +copies live, *how* they're encrypted, or *whether restores actually work*. +`CAPABILITIES.md` §9 sketches an intent (PBS + restic, pCloud off-site, USB air-gap) +but commits to nothing. + +This design defines the policy end-to-end: recovery model, what is captured and how, +the 3-2-1 topology, encryption and key escrow with a break-glass path, restore +testing, retention, failure alerting, and the air-gap mechanism. + +## Scope + +- **In:** application *state* backup for boma's hosts and services; off-site and + air-gapped copies; encryption + key escrow; restore testing; failure alerting; + retention; the backup node. +- **Out (for now):** whole-VM image backup (Proxmox Backup Server) — explicitly + deferred, see Decision 1; a central-vs-per-app database decision (TODO 3.9 — this + design is agnostic to it); Prometheus backup metrics (noted as a later add). + +## Decisions (as settled) + +### 1. Recovery model — data-only backups, rebuild from code (Model A) + +boma's *configuration* is reproducible from this repo: Terraform recreates the VM, +Ansible re-renders the Docker Compose stack. So backups protect **state only** — DB +contents, bind-mount data dirs, Vaultwarden's vault — not whole-VM images. + +To recover a host: Terraform re-provisions the VM → Ansible redeploys → restic +restores the data. **No Proxmox Backup Server.** This keeps 3-2-1 cheap, fits +pCloud's 1 TB comfortably, and turns every restore into a continuous proof that the +IaC *and* the backups both work. + +Trade-off accepted: recovery is slower than a VM-image restore (a full Ansible run + +data restore, potentially hours), and it bets the repo is complete enough to rebuild +from nothing — which Tier-2 restore testing (Decision 8) exists to verify. **PBS +(Model B) or a per-host hybrid (Model C) can be added later** if real-world RTO proves +too slow; nothing here precludes it. + +### 2. One backup tier, ~24 h RPO + +A single tier: nightly backup of all state, accepting up to ~24 h of data loss across +the board. No per-data-type tiering yet — revisit once there is real-world data and +experience to justify the added machinery. + +### 3. Engine — restic (data) + rclone (off-site); no PBS + +- **restic** captures state into an encrypted, deduplicated repository. +- **rclone** replicates the repo to pCloud (pCloud has no good headless Linux client; + rclone has a first-class pCloud backend). +- restic encrypts the repo at rest, so rclone copies **ciphertext only** — no second + encryption layer, no pCloud "crypto folder." + +### 4. Topology — central pull node (`fisi`), off the cluster + +A single backup node owns the canonical restic repo. It is **off the Proxmox +cluster** — an independent failure domain, so copy 2 survives a PVE node (or the whole +cluster) dying. This mirrors the existing pattern for `ubongo` (control) and `askari` +(off-site): a manually-provisioned physical node in its own inventory group, still +Ansible-managed (base hardening + a `backup` role). + +**Pull model.** The backup node holds SSH keys to each host; per service it runs the +declared dump command remotely, pulls the declared paths read-only, then `restic` +snapshots the staged data into its *local* repo. **Hosts hold no backup credentials +and cannot reach the repo** — so a compromised or ransomwared service host cannot +delete backup history. + +**Backup node assignment:** `fisi` (an HP Elite 600 G9 tower), penciled in / provisional +— the *role* ("the backup node") is load-bearing; the physical assignment may be +revisited when all hardware is on hand. `fisi` holds **2× 8 TB HDDs in a mirror** +(ZFS or mdraid → 8 TB usable, survives one disk failure; not a stripe). It owns the +repo, runs the pull orchestration, runs `rclone → pCloud`, and **docks the USB +air-gap drives** (Decision 11). Pending one hardware item: the SATA power cable from +the board/PSU to the drives. A data-only restic node is a featherweight workload, so +the G9 is comfortably over-specced. + +### 5. 3-2-1 mapping + +| Copy | Location | Medium | Off-site | Notes | +|---|---|---|---|---| +| 1 | Live data on each host | NVMe/SSD | no | The working data | +| 2 | `fisi` restic repo | 8 TB HDD mirror | no (on-site, off-cluster) | Canonical repo | +| 3 | pCloud (via rclone) | Cloud | **yes** | Encrypted ciphertext; **sync-coupled** (see Decision 9 / threat model) | +| +4 | USB air-gap drive(s) | Removable HDD, **offline** | yes (stored off-site) | The **immutable backstop**; rotated | + +≥3 copies, ≥2 media, ≥1 off-site — satisfied, with the air-gap drive as a fourth, +offline copy that no online compromise can reach. + +### 6. Per-service backup contract — `backup__*` data + `BACKUP.md` (hard convention) + +Almost every boma service is the same shape: a Docker bind-mount data dir + maybe a +database. Each **service role declares its backup needs** in role vars — the same +render-from-data pattern boma uses for `access__*`/`ACCESS.md` (ADR-021): + +```yaml +backup__paths: # bind-mount dirs / files holding state + - /srv/nextcloud/data +backup__predump: # optional: command that emits an app-consistent dump + cmd: "docker compose exec -T db pg_dump -U {{ ... }} nextcloud" + dest: "nextcloud-db.sql" +``` + +The pull orchestrator reads these (rendered from inventory) and, per service: SSH in → +run `predump` → pull the dump + declared paths read-only → `restic` snapshot. A +service with **no** `backup__paths` is explicitly "nothing to back up" (declared, not +silent). + +**`BACKUP.md` becomes a required per-service doc** alongside `SECURITY.md` / +`VERIFY.md` / `ACCESS.md`, **rendered from the role's `backup__*` data**, documenting: +what state exists, what is backed up, the dump command, and the per-service **restore** +procedure. A template lives at `docs/backup/service-backup-template.md`. `make lint` +gates its presence for service roles. + +### 7. Consistency — logical dumps first, quiesce as an escape hatch + +- **Default (A):** databases are captured with logical dumps (`pg_dump` / + `mysqldump`) — portable, version-independent, restorable to a fresh DB. Plain data + dirs are backed up as files. No downtime. Cost: every stateful service must declare + a working dump command, *tested by restore drills*. +- **Escape hatch (B):** a service whose data cannot be dumped live declares a + quiesce step (stop container → back up volume → restart) in the same contract. +- ZFS/filesystem snapshots are **not** used as the sole DB method (only + crash-consistent for a live database). + +This is agnostic to the open central-vs-per-app database question (TODO 3.9): either +way, each service declares how to dump its own data. + +### 8. Restore testing — two tiers + +- **Tier 1 — frequent, automated, rolling restore-verify (weekly).** Pick the next + service in rotation, restore its latest snapshot into a throwaway **container on + `ubongo`** (reusing boma's existing Molecule harness, ADR-015), start the app + against the restored data, and **run that service's `VERIFY.md` checks** + (ADR-008/017) against it, then tear down. This catches the failure that actually + kills people — *silently corrupt or unrestorable backups*. Failures alert via ntfy. +- **Tier 2 — rare, full DR rehearsal (semi-annual), driven from `ubongo` onto PVE + staging.** Rebuild a host from zero via Terraform + Ansible + restic restore on the + staging cluster (only a real PVE node can host the VM; `ubongo` orchestrates). This + validates the whole Model-A recovery chain, not just "can I read a snapshot." + **At least once a year the rehearsal exercises the paper-secret break-glass path** + (Decision 10) end-to-end. + +`ubongo` stays **bare Debian, not a hypervisor** (ADR-015 unchanged): its job is to be +the independent recovery anchor — "the tool used to rebuild the cluster must not live +inside the thing it rebuilds." Higher-fidelity real-VM testing is *better* served by +the PVE staging env (same hardware class, same cluster, same provisioning path) than +by converting `ubongo`. `ubongo`'s real spec is a ThinkCentre M70q (i3-10100T / 16 GB +/ **1 TB NVMe**) — the 1 TB gives ample room for Tier-1 dataset restores; disk +headroom (not CPU/RAM) is the first thing to watch as data grows (`/capacity-review`). + +### 9. Retention — GFS via restic + +Starting policy: `--keep-daily 7 --keep-weekly 4 --keep-monthly 6 --keep-yearly 1`. +`restic forget --prune` runs nightly on `fisi`'s repo; pCloud mirrors the pruned repo. +Tune once real repo growth is observed. + +### 10. Encryption + key escrow + break-glass + +restic already encrypts the repo, so **one secret — the restic repo password — +protects all copies uniformly** (fisi, pCloud, USB). One thing to escrow, not three. + +**Escrow locations:** +- **`fisi`, root-only** (+ in the Ansible vault) — so backups run non-interactively + and `fisi` is redeployable. +- **Vaultwarden** — the day-to-day human-accessible copy. +- **Paper, in a physical safe (off-site)** — the break-glass root of trust; the only + copy that survives "everything is down." + +**Model-A twist — the paper holds *two* secrets, not one:** +1. the **restic repo password** (to read any backup at all), and +2. the **Ansible vault master password** (to rebuild hosts from the repo — normally + from Vaultwarden via `rbw`, which is itself down in a from-zero recovery). + +With both on paper, the break-glass chain has **no circular dependency**: paper → +restic restores Vaultwarden + repo data → the vault password (from paper) drives +Terraform/Ansible re-provisioning → services return, `rbw` works again. `ubongo`'s +ADR-015 recovery model already establishes **`mamba` (laptop) as a break-glass clone** +(repo + toolchain + mesh + `rbw`, with Terraform state synced to it) — the rebuild can +be driven from `mamba` if `ubongo` is also gone. The printed sheet is a short +**break-glass runbook** assuming zero running boma infrastructure: install restic on +any machine, point it at pCloud *or* a USB drive with the password, restore Vaultwarden +first, then rebuild with the vault password. + +### 11. USB air-gap trigger (plug-and-go cold copy) + +A **udev rule on `fisi` matching an allowlist of known drive serials** triggers a +systemd unit → script that: mounts the drive, confirms it is an expected drive, runs +**`restic copy` from the local repo → a restic repo on the USB drive** (dedup-aware, +same password → ciphertext if lost/stolen), runs `restic check` on the USB copy, +unmounts, and **notifies via ntfy** with the result. Only allowlisted serials trigger +anything (a rogue USB does nothing). + +`restic copy` (not rsync) so the USB is itself a valid restic repo — restorable +**directly** in a break-glass with nothing else alive. Rotate among a few drives, +**stored off-site** → also a second *geographic* off-site copy independent of pCloud. + +### 12. Failure alerting — guard against silent death + +Success/failure pings alone miss the worst case (*the job silently stopped running*): +- **Dead-man's-switch:** every successful nightly run pings an **Uptime Kuma push + monitor** (already in the planned stack); no ping in ~25 h → alert. +- **Immediate failure → ntfy** on any job or `predump` error. +- **Periodic `restic check`** (weekly) for repo integrity → alert on corruption. +- **Tier-1 restore-verify failures → ntfy.** +- *(Later)* emit last-success timestamp + repo size as Prometheus metrics for a + Grafana panel (fits ADR-018's monitoring direction; not required for v1). + +### 13. Schedule + +- **Nightly backup run (~02:00–04:00),** driven by `fisi` (pull): per host → + `predump` → pull paths read-only → `restic` snapshot → `restic forget --prune` + (Decision 9) → `rclone sync` → pCloud. Sequential, off-hours. +- **Tier-1 restore-verify:** weekly, rolling one service, on `ubongo`. +- **Tier-2 DR rehearsal:** semi-annual on staging; ≥1/year exercises the paper path. +- **USB air-gap:** manual, ~monthly, whenever a drive is docked. + +## Architecture & data flow (nightly run) + +``` + ┌─────────────────────────────────────────┐ + docker_hosts / etc. │ fisi (backup node) │ + ┌───────────┐ SSH │ pull orchestrator (reads backup__* ) │ + │ service A │◀─────────│ 1. ssh host → run predump (pg_dump…) │ + │ + DB │ pull RO │ 2. pull dump + backup__paths (read-only)│ + └───────────┘─────────▶│ 3. restic snapshot → local repo (mirror)│ + ┌───────────┐ │ 4. restic forget --prune (GFS) │ + │ service B │ │ 5. rclone sync repo → pCloud (offsite) │ + └───────────┘ │ 6. heartbeat → Uptime Kuma; errors→ntfy│ + └───────────────┬──────────────────────────┘ + │ (manual, ~monthly) + udev: known drive plugged + ▼ + restic copy → USB repo (air-gap, offline) +``` + +Restore (Model A): Terraform re-provisions the VM → Ansible redeploys the role → +restic restores `backup__paths` + replays the dump → `VERIFY.md` confirms. + +## Components & boundaries + +- **`backup` role (on `fisi`):** pull orchestrator, restic repo management, retention + prune, rclone→pCloud sync, udev/air-gap unit, alerting hooks. New inventory group + (e.g. `backup_hosts`) with the `base` role applied, like `control`/`offsite_hosts`. +- **Per-service backup contract:** `backup__*` role vars + rendered `BACKUP.md` + (Decision 6); a hard convention enforced by `make lint`. +- **`ubongo`:** schedules/drives Tier-1 (local container) and Tier-2 (onto staging); + unchanged role per ADR-015. +- **Secrets:** restic password + rclone token in `fisi` (root-only) and the Ansible + vault; escrowed per Decision 10. + +## Threat model / 3-2-1 honesty + +- **`rclone sync` propagates deletions** — a prune, or a *malicious* wipe of `fisi`'s + repo, replicates to pCloud. pCloud is therefore the **off-site** copy but **not + immutable**. Mitigations: the **USB air-gap drive is the immutable backstop** + (offline = unreachable by any online compromise) and **pCloud's own file-version + history** is enabled as a recovery cushion. +- **Pull model** stops a compromised *service host* from touching the repo. +- **`fisi` is the crown-jewel host** — it holds an encrypted copy of all state, so it + gets full base hardening and tight access. restic encryption means a stolen `fisi` + (or USB, or pCloud blob) yields ciphertext only. +- **pCloud's 1 TB is the smallest copy → the off-site capacity ceiling.** Data-only + backups fit for years at homelab scale; flag for `/capacity-review` if the repo + trends toward ~1 TB. + +## What this changes in the repo (for the plan) + +- New `backup` role + `backup_hosts` inventory group; `fisi` hardware-reference entry. +- New per-service convention: `backup__*` vars + `BACKUP.md` (template at + `docs/backup/service-backup-template.md`); `make lint` gate; update role-conventions + in `CLAUDE.md` and the new-role scaffolding/runbook. +- Update `docs/hardware/reference.md`: `ubongo` = M70q (i3-10100T/16 GB/**1 TB**); + add `fisi`. +- Update `CAPABILITIES.md` §9 (PBS → deferred; restic+rclone+USB the committed engine). +- Close `docs/TODO.md` 3.8; cross-reference from ADR-011. +- The break-glass runbook (printed sheet + `docs/runbooks/`), referencing ADR-015's + `mamba` clone and Terraform-state survival. + +## Non-goals / YAGNI + +- No PBS / whole-VM images in v1 (Decision 1). +- No per-data-type RPO tiering in v1 (Decision 2). +- No second encryption layer over restic (Decision 3). +- No central NAS/file-share scope creep on `fisi` — it stays single-purpose. + +## Open / deferred + +- Central vs per-app database (TODO 3.9) — orthogonal; this design works either way. +- Prometheus backup metrics — later add (Decision 12). +- PBS (Model B) or hybrid (Model C) — revisit if real-world RTO is too slow.