docs(backup): add backup & DR strategy design (→ ADR-022)
Data-only restic backups, rebuild-from-code recovery (Model A); central off-cluster pull node (fisi) with 8TB mirror; 3-2-1 via pCloud (rclone) + rotated USB air-gap. Per-service backup__* contract + BACKUP.md as a hard convention. Two-tier restore testing (ubongo container restore-verify + semi-annual staging DR rehearsal). One restic password escrowed to Vaultwarden + paper (restic + vault passwords) for a non-circular break-glass. Dead-man's-switch alerting via Uptime Kuma. Resolves TODO 3.8; grounds ADR-011's backup-first assumption. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
032adf1525
commit
eaffd8d900
1 changed files with 310 additions and 0 deletions
310
docs/superpowers/specs/2026-06-10-backup-strategy-design.md
Normal file
310
docs/superpowers/specs/2026-06-10-backup-strategy-design.md
Normal file
|
|
@ -0,0 +1,310 @@
|
|||
# Design — Backup & disaster recovery strategy
|
||||
|
||||
- **Date:** 2026-06-10
|
||||
- **Status:** Approved design — pending implementation plan
|
||||
- **Resolves:** `docs/TODO.md` item 3.8 ("ensure the right things are backed up,
|
||||
incl. DB dumps") and `docs/CAPABILITIES.md` §9 (backup engine / off-site / air-gap,
|
||||
all "planned")
|
||||
- **Grounds:** the backup substrate that ADR-011 (update management) already leans on
|
||||
("snapshot-before + backups remain the rollback mechanism", "always dumps the DB /
|
||||
takes a backup first") but never defined
|
||||
- **Reuses:** ADR-004 (one service = one role; per-service doc conventions),
|
||||
ADR-008/017 (`VERIFY.md` per-service checks), ADR-021 (`ACCESS.md` rendered from
|
||||
role `access__*` data — the same render-from-data pattern), ADR-015 (`ubongo`
|
||||
recovery model; `mamba` break-glass clone)
|
||||
- **Becomes:** ADR-022 (this design is the basis for that ADR)
|
||||
|
||||
---
|
||||
|
||||
## Problem
|
||||
|
||||
boma has no defined backup policy. The ADRs assume one exists — ADR-011 makes
|
||||
"backup-first" the rule for stateful upgrades and "snapshot + backup" the rollback
|
||||
path — but nothing specifies *what* gets backed up, *how* it stays consistent, *where*
|
||||
copies live, *how* they're encrypted, or *whether restores actually work*.
|
||||
`CAPABILITIES.md` §9 sketches an intent (PBS + restic, pCloud off-site, USB air-gap)
|
||||
but commits to nothing.
|
||||
|
||||
This design defines the policy end-to-end: recovery model, what is captured and how,
|
||||
the 3-2-1 topology, encryption and key escrow with a break-glass path, restore
|
||||
testing, retention, failure alerting, and the air-gap mechanism.
|
||||
|
||||
## Scope
|
||||
|
||||
- **In:** application *state* backup for boma's hosts and services; off-site and
|
||||
air-gapped copies; encryption + key escrow; restore testing; failure alerting;
|
||||
retention; the backup node.
|
||||
- **Out (for now):** whole-VM image backup (Proxmox Backup Server) — explicitly
|
||||
deferred, see Decision 1; a central-vs-per-app database decision (TODO 3.9 — this
|
||||
design is agnostic to it); Prometheus backup metrics (noted as a later add).
|
||||
|
||||
## Decisions (as settled)
|
||||
|
||||
### 1. Recovery model — data-only backups, rebuild from code (Model A)
|
||||
|
||||
boma's *configuration* is reproducible from this repo: Terraform recreates the VM,
|
||||
Ansible re-renders the Docker Compose stack. So backups protect **state only** — DB
|
||||
contents, bind-mount data dirs, Vaultwarden's vault — not whole-VM images.
|
||||
|
||||
To recover a host: Terraform re-provisions the VM → Ansible redeploys → restic
|
||||
restores the data. **No Proxmox Backup Server.** This keeps 3-2-1 cheap, fits
|
||||
pCloud's 1 TB comfortably, and turns every restore into a continuous proof that the
|
||||
IaC *and* the backups both work.
|
||||
|
||||
Trade-off accepted: recovery is slower than a VM-image restore (a full Ansible run +
|
||||
data restore, potentially hours), and it bets the repo is complete enough to rebuild
|
||||
from nothing — which Tier-2 restore testing (Decision 8) exists to verify. **PBS
|
||||
(Model B) or a per-host hybrid (Model C) can be added later** if real-world RTO proves
|
||||
too slow; nothing here precludes it.
|
||||
|
||||
### 2. One backup tier, ~24 h RPO
|
||||
|
||||
A single tier: nightly backup of all state, accepting up to ~24 h of data loss across
|
||||
the board. No per-data-type tiering yet — revisit once there is real-world data and
|
||||
experience to justify the added machinery.
|
||||
|
||||
### 3. Engine — restic (data) + rclone (off-site); no PBS
|
||||
|
||||
- **restic** captures state into an encrypted, deduplicated repository.
|
||||
- **rclone** replicates the repo to pCloud (pCloud has no good headless Linux client;
|
||||
rclone has a first-class pCloud backend).
|
||||
- restic encrypts the repo at rest, so rclone copies **ciphertext only** — no second
|
||||
encryption layer, no pCloud "crypto folder."
|
||||
|
||||
### 4. Topology — central pull node (`fisi`), off the cluster
|
||||
|
||||
A single backup node owns the canonical restic repo. It is **off the Proxmox
|
||||
cluster** — an independent failure domain, so copy 2 survives a PVE node (or the whole
|
||||
cluster) dying. This mirrors the existing pattern for `ubongo` (control) and `askari`
|
||||
(off-site): a manually-provisioned physical node in its own inventory group, still
|
||||
Ansible-managed (base hardening + a `backup` role).
|
||||
|
||||
**Pull model.** The backup node holds SSH keys to each host; per service it runs the
|
||||
declared dump command remotely, pulls the declared paths read-only, then `restic`
|
||||
snapshots the staged data into its *local* repo. **Hosts hold no backup credentials
|
||||
and cannot reach the repo** — so a compromised or ransomwared service host cannot
|
||||
delete backup history.
|
||||
|
||||
**Backup node assignment:** `fisi` (an HP Elite 600 G9 tower), penciled in / provisional
|
||||
— the *role* ("the backup node") is load-bearing; the physical assignment may be
|
||||
revisited when all hardware is on hand. `fisi` holds **2× 8 TB HDDs in a mirror**
|
||||
(ZFS or mdraid → 8 TB usable, survives one disk failure; not a stripe). It owns the
|
||||
repo, runs the pull orchestration, runs `rclone → pCloud`, and **docks the USB
|
||||
air-gap drives** (Decision 11). Pending one hardware item: the SATA power cable from
|
||||
the board/PSU to the drives. A data-only restic node is a featherweight workload, so
|
||||
the G9 is comfortably over-specced.
|
||||
|
||||
### 5. 3-2-1 mapping
|
||||
|
||||
| Copy | Location | Medium | Off-site | Notes |
|
||||
|---|---|---|---|---|
|
||||
| 1 | Live data on each host | NVMe/SSD | no | The working data |
|
||||
| 2 | `fisi` restic repo | 8 TB HDD mirror | no (on-site, off-cluster) | Canonical repo |
|
||||
| 3 | pCloud (via rclone) | Cloud | **yes** | Encrypted ciphertext; **sync-coupled** (see Decision 9 / threat model) |
|
||||
| +4 | USB air-gap drive(s) | Removable HDD, **offline** | yes (stored off-site) | The **immutable backstop**; rotated |
|
||||
|
||||
≥3 copies, ≥2 media, ≥1 off-site — satisfied, with the air-gap drive as a fourth,
|
||||
offline copy that no online compromise can reach.
|
||||
|
||||
### 6. Per-service backup contract — `backup__*` data + `BACKUP.md` (hard convention)
|
||||
|
||||
Almost every boma service is the same shape: a Docker bind-mount data dir + maybe a
|
||||
database. Each **service role declares its backup needs** in role vars — the same
|
||||
render-from-data pattern boma uses for `access__*`/`ACCESS.md` (ADR-021):
|
||||
|
||||
```yaml
|
||||
backup__paths: # bind-mount dirs / files holding state
|
||||
- /srv/nextcloud/data
|
||||
backup__predump: # optional: command that emits an app-consistent dump
|
||||
cmd: "docker compose exec -T db pg_dump -U {{ ... }} nextcloud"
|
||||
dest: "nextcloud-db.sql"
|
||||
```
|
||||
|
||||
The pull orchestrator reads these (rendered from inventory) and, per service: SSH in →
|
||||
run `predump` → pull the dump + declared paths read-only → `restic` snapshot. A
|
||||
service with **no** `backup__paths` is explicitly "nothing to back up" (declared, not
|
||||
silent).
|
||||
|
||||
**`BACKUP.md` becomes a required per-service doc** alongside `SECURITY.md` /
|
||||
`VERIFY.md` / `ACCESS.md`, **rendered from the role's `backup__*` data**, documenting:
|
||||
what state exists, what is backed up, the dump command, and the per-service **restore**
|
||||
procedure. A template lives at `docs/backup/service-backup-template.md`. `make lint`
|
||||
gates its presence for service roles.
|
||||
|
||||
### 7. Consistency — logical dumps first, quiesce as an escape hatch
|
||||
|
||||
- **Default (A):** databases are captured with logical dumps (`pg_dump` /
|
||||
`mysqldump`) — portable, version-independent, restorable to a fresh DB. Plain data
|
||||
dirs are backed up as files. No downtime. Cost: every stateful service must declare
|
||||
a working dump command, *tested by restore drills*.
|
||||
- **Escape hatch (B):** a service whose data cannot be dumped live declares a
|
||||
quiesce step (stop container → back up volume → restart) in the same contract.
|
||||
- ZFS/filesystem snapshots are **not** used as the sole DB method (only
|
||||
crash-consistent for a live database).
|
||||
|
||||
This is agnostic to the open central-vs-per-app database question (TODO 3.9): either
|
||||
way, each service declares how to dump its own data.
|
||||
|
||||
### 8. Restore testing — two tiers
|
||||
|
||||
- **Tier 1 — frequent, automated, rolling restore-verify (weekly).** Pick the next
|
||||
service in rotation, restore its latest snapshot into a throwaway **container on
|
||||
`ubongo`** (reusing boma's existing Molecule harness, ADR-015), start the app
|
||||
against the restored data, and **run that service's `VERIFY.md` checks**
|
||||
(ADR-008/017) against it, then tear down. This catches the failure that actually
|
||||
kills people — *silently corrupt or unrestorable backups*. Failures alert via ntfy.
|
||||
- **Tier 2 — rare, full DR rehearsal (semi-annual), driven from `ubongo` onto PVE
|
||||
staging.** Rebuild a host from zero via Terraform + Ansible + restic restore on the
|
||||
staging cluster (only a real PVE node can host the VM; `ubongo` orchestrates). This
|
||||
validates the whole Model-A recovery chain, not just "can I read a snapshot."
|
||||
**At least once a year the rehearsal exercises the paper-secret break-glass path**
|
||||
(Decision 10) end-to-end.
|
||||
|
||||
`ubongo` stays **bare Debian, not a hypervisor** (ADR-015 unchanged): its job is to be
|
||||
the independent recovery anchor — "the tool used to rebuild the cluster must not live
|
||||
inside the thing it rebuilds." Higher-fidelity real-VM testing is *better* served by
|
||||
the PVE staging env (same hardware class, same cluster, same provisioning path) than
|
||||
by converting `ubongo`. `ubongo`'s real spec is a ThinkCentre M70q (i3-10100T / 16 GB
|
||||
/ **1 TB NVMe**) — the 1 TB gives ample room for Tier-1 dataset restores; disk
|
||||
headroom (not CPU/RAM) is the first thing to watch as data grows (`/capacity-review`).
|
||||
|
||||
### 9. Retention — GFS via restic
|
||||
|
||||
Starting policy: `--keep-daily 7 --keep-weekly 4 --keep-monthly 6 --keep-yearly 1`.
|
||||
`restic forget --prune` runs nightly on `fisi`'s repo; pCloud mirrors the pruned repo.
|
||||
Tune once real repo growth is observed.
|
||||
|
||||
### 10. Encryption + key escrow + break-glass
|
||||
|
||||
restic already encrypts the repo, so **one secret — the restic repo password —
|
||||
protects all copies uniformly** (fisi, pCloud, USB). One thing to escrow, not three.
|
||||
|
||||
**Escrow locations:**
|
||||
- **`fisi`, root-only** (+ in the Ansible vault) — so backups run non-interactively
|
||||
and `fisi` is redeployable.
|
||||
- **Vaultwarden** — the day-to-day human-accessible copy.
|
||||
- **Paper, in a physical safe (off-site)** — the break-glass root of trust; the only
|
||||
copy that survives "everything is down."
|
||||
|
||||
**Model-A twist — the paper holds *two* secrets, not one:**
|
||||
1. the **restic repo password** (to read any backup at all), and
|
||||
2. the **Ansible vault master password** (to rebuild hosts from the repo — normally
|
||||
from Vaultwarden via `rbw`, which is itself down in a from-zero recovery).
|
||||
|
||||
With both on paper, the break-glass chain has **no circular dependency**: paper →
|
||||
restic restores Vaultwarden + repo data → the vault password (from paper) drives
|
||||
Terraform/Ansible re-provisioning → services return, `rbw` works again. `ubongo`'s
|
||||
ADR-015 recovery model already establishes **`mamba` (laptop) as a break-glass clone**
|
||||
(repo + toolchain + mesh + `rbw`, with Terraform state synced to it) — the rebuild can
|
||||
be driven from `mamba` if `ubongo` is also gone. The printed sheet is a short
|
||||
**break-glass runbook** assuming zero running boma infrastructure: install restic on
|
||||
any machine, point it at pCloud *or* a USB drive with the password, restore Vaultwarden
|
||||
first, then rebuild with the vault password.
|
||||
|
||||
### 11. USB air-gap trigger (plug-and-go cold copy)
|
||||
|
||||
A **udev rule on `fisi` matching an allowlist of known drive serials** triggers a
|
||||
systemd unit → script that: mounts the drive, confirms it is an expected drive, runs
|
||||
**`restic copy` from the local repo → a restic repo on the USB drive** (dedup-aware,
|
||||
same password → ciphertext if lost/stolen), runs `restic check` on the USB copy,
|
||||
unmounts, and **notifies via ntfy** with the result. Only allowlisted serials trigger
|
||||
anything (a rogue USB does nothing).
|
||||
|
||||
`restic copy` (not rsync) so the USB is itself a valid restic repo — restorable
|
||||
**directly** in a break-glass with nothing else alive. Rotate among a few drives,
|
||||
**stored off-site** → also a second *geographic* off-site copy independent of pCloud.
|
||||
|
||||
### 12. Failure alerting — guard against silent death
|
||||
|
||||
Success/failure pings alone miss the worst case (*the job silently stopped running*):
|
||||
- **Dead-man's-switch:** every successful nightly run pings an **Uptime Kuma push
|
||||
monitor** (already in the planned stack); no ping in ~25 h → alert.
|
||||
- **Immediate failure → ntfy** on any job or `predump` error.
|
||||
- **Periodic `restic check`** (weekly) for repo integrity → alert on corruption.
|
||||
- **Tier-1 restore-verify failures → ntfy.**
|
||||
- *(Later)* emit last-success timestamp + repo size as Prometheus metrics for a
|
||||
Grafana panel (fits ADR-018's monitoring direction; not required for v1).
|
||||
|
||||
### 13. Schedule
|
||||
|
||||
- **Nightly backup run (~02:00–04:00),** driven by `fisi` (pull): per host →
|
||||
`predump` → pull paths read-only → `restic` snapshot → `restic forget --prune`
|
||||
(Decision 9) → `rclone sync` → pCloud. Sequential, off-hours.
|
||||
- **Tier-1 restore-verify:** weekly, rolling one service, on `ubongo`.
|
||||
- **Tier-2 DR rehearsal:** semi-annual on staging; ≥1/year exercises the paper path.
|
||||
- **USB air-gap:** manual, ~monthly, whenever a drive is docked.
|
||||
|
||||
## Architecture & data flow (nightly run)
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────┐
|
||||
docker_hosts / etc. │ fisi (backup node) │
|
||||
┌───────────┐ SSH │ pull orchestrator (reads backup__* ) │
|
||||
│ service A │◀─────────│ 1. ssh host → run predump (pg_dump…) │
|
||||
│ + DB │ pull RO │ 2. pull dump + backup__paths (read-only)│
|
||||
└───────────┘─────────▶│ 3. restic snapshot → local repo (mirror)│
|
||||
┌───────────┐ │ 4. restic forget --prune (GFS) │
|
||||
│ service B │ │ 5. rclone sync repo → pCloud (offsite) │
|
||||
└───────────┘ │ 6. heartbeat → Uptime Kuma; errors→ntfy│
|
||||
└───────────────┬──────────────────────────┘
|
||||
│ (manual, ~monthly)
|
||||
udev: known drive plugged
|
||||
▼
|
||||
restic copy → USB repo (air-gap, offline)
|
||||
```
|
||||
|
||||
Restore (Model A): Terraform re-provisions the VM → Ansible redeploys the role →
|
||||
restic restores `backup__paths` + replays the dump → `VERIFY.md` confirms.
|
||||
|
||||
## Components & boundaries
|
||||
|
||||
- **`backup` role (on `fisi`):** pull orchestrator, restic repo management, retention
|
||||
prune, rclone→pCloud sync, udev/air-gap unit, alerting hooks. New inventory group
|
||||
(e.g. `backup_hosts`) with the `base` role applied, like `control`/`offsite_hosts`.
|
||||
- **Per-service backup contract:** `backup__*` role vars + rendered `BACKUP.md`
|
||||
(Decision 6); a hard convention enforced by `make lint`.
|
||||
- **`ubongo`:** schedules/drives Tier-1 (local container) and Tier-2 (onto staging);
|
||||
unchanged role per ADR-015.
|
||||
- **Secrets:** restic password + rclone token in `fisi` (root-only) and the Ansible
|
||||
vault; escrowed per Decision 10.
|
||||
|
||||
## Threat model / 3-2-1 honesty
|
||||
|
||||
- **`rclone sync` propagates deletions** — a prune, or a *malicious* wipe of `fisi`'s
|
||||
repo, replicates to pCloud. pCloud is therefore the **off-site** copy but **not
|
||||
immutable**. Mitigations: the **USB air-gap drive is the immutable backstop**
|
||||
(offline = unreachable by any online compromise) and **pCloud's own file-version
|
||||
history** is enabled as a recovery cushion.
|
||||
- **Pull model** stops a compromised *service host* from touching the repo.
|
||||
- **`fisi` is the crown-jewel host** — it holds an encrypted copy of all state, so it
|
||||
gets full base hardening and tight access. restic encryption means a stolen `fisi`
|
||||
(or USB, or pCloud blob) yields ciphertext only.
|
||||
- **pCloud's 1 TB is the smallest copy → the off-site capacity ceiling.** Data-only
|
||||
backups fit for years at homelab scale; flag for `/capacity-review` if the repo
|
||||
trends toward ~1 TB.
|
||||
|
||||
## What this changes in the repo (for the plan)
|
||||
|
||||
- New `backup` role + `backup_hosts` inventory group; `fisi` hardware-reference entry.
|
||||
- New per-service convention: `backup__*` vars + `BACKUP.md` (template at
|
||||
`docs/backup/service-backup-template.md`); `make lint` gate; update role-conventions
|
||||
in `CLAUDE.md` and the new-role scaffolding/runbook.
|
||||
- Update `docs/hardware/reference.md`: `ubongo` = M70q (i3-10100T/16 GB/**1 TB**);
|
||||
add `fisi`.
|
||||
- Update `CAPABILITIES.md` §9 (PBS → deferred; restic+rclone+USB the committed engine).
|
||||
- Close `docs/TODO.md` 3.8; cross-reference from ADR-011.
|
||||
- The break-glass runbook (printed sheet + `docs/runbooks/`), referencing ADR-015's
|
||||
`mamba` clone and Terraform-state survival.
|
||||
|
||||
## Non-goals / YAGNI
|
||||
|
||||
- No PBS / whole-VM images in v1 (Decision 1).
|
||||
- No per-data-type RPO tiering in v1 (Decision 2).
|
||||
- No second encryption layer over restic (Decision 3).
|
||||
- No central NAS/file-share scope creep on `fisi` — it stays single-purpose.
|
||||
|
||||
## Open / deferred
|
||||
|
||||
- Central vs per-app database (TODO 3.9) — orthogonal; this design works either way.
|
||||
- Prometheus backup metrics — later add (Decision 12).
|
||||
- PBS (Model B) or hybrid (Model C) — revisit if real-world RTO is too slow.
|
||||
Loading…
Add table
Reference in a new issue