sjat
b3ca510380
010/011: relabel Decisions->Decision + add Status/Consequences. 013: add Status + Decision umbrella (existing Consequences untouched). No decision substance changed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
154 lines
8.4 KiB
Markdown
154 lines
8.4 KiB
Markdown
# ADR-011 — Update and upgrade management
|
||
|
||
## Status
|
||
|
||
Accepted (2026-06-04)
|
||
|
||
**Status: Proposed — draft for discussion (not yet accepted).**
|
||
|
||
## Context
|
||
|
||
boma runs Debian 13 VMs, each hosting a set of Docker Compose services. Two things
|
||
drift over time and must be kept current without breaking the homelab: the **host OS**
|
||
(kernel, libc, packages → sometimes a reboot) and the **container images**.
|
||
|
||
---
|
||
|
||
## Decision
|
||
|
||
### 1. Every service is classified stateful or stateless
|
||
|
||
Each container role declares its class, e.g. `<role>__stateful: true|false` (default
|
||
`false`). The split is the load-bearing classification for the whole policy.
|
||
|
||
- **Stateless** — no durable data of its own; losing the container loses nothing.
|
||
Rebuild = re-pull. Examples: the \*arr stack, Jellyfin, exporters, whoami, Traefik,
|
||
reverse proxies, FlareSolverr.
|
||
- **Stateful** — owns data, schema, or migrations: databases, and apps with their own
|
||
store/migrations (Nextcloud, Vaultwarden, Forgejo, PhotoPrism, Discourse, Snipe-IT).
|
||
When in doubt, classify **stateful** (the safer, slower path).
|
||
|
||
### 2. Image pinning follows the split
|
||
|
||
- **Stateless → rolling tags** (`latest`/`stable`), refreshed by the weekly run and
|
||
watched by DIUN. Always-current, cheap to roll back. No digest pin — it would
|
||
defeat the rolling design.
|
||
- **Stateful → pinned `tag@digest`** — a readable **minor** tag where the image
|
||
offers it (e.g. `mariadb:11.4`, not bare `:11`) **plus its digest**
|
||
(`mariadb:11.4@sha256:…`). Reproducible and tamper-evident; upgrades are deliberate
|
||
(bump tag and digest together), never incidental.
|
||
|
||
Readable tag **and** digest, not one or the other: the tag keeps diffs legible, the
|
||
digest pins the exact bytes for supply-chain integrity (ADR-002, accepted-risk R1).
|
||
Snapshot-before + backups remain the rollback mechanism for a *broken* update; the
|
||
digest is what guards against a *swapped* image, which snapshots cannot.
|
||
|
||
### 3. Weekly OS + stateless run — Friday night, fail-stop, staggered
|
||
|
||
A scheduled run on **Friday night** (giving the weekend to fix anything it breaks),
|
||
per host, in strict order with a verification gate between every phase:
|
||
|
||
1. **OS update** — `apt` upgrade.
|
||
2. **Reboot** — only if required (kernel/libc); detect via `/var/run/reboot-required`.
|
||
3. **Verify** — health-check harness. **Fail-stop:** if a host fails,
|
||
halt _that host's_ run, leave it as-is, alert loudly — do **not** proceed
|
||
to container updates on a wobbly host.
|
||
4. **Stateless container update** — `compose pull` + recreate-if-changed.
|
||
5. **Verify** again; alert on failure.
|
||
|
||
**Host ordering:** infrastructure hosts (DNS, then reverse proxy) update and validate
|
||
**before** the rest follow — so a DNS/Traefik failure doesn't make every host look
|
||
broken at once and hide the real cause. Never reboot the whole fleet simultaneously.
|
||
|
||
### 4. Snapshot-before is the rollback mechanism
|
||
|
||
Because these are primarily Proxmox VMs, take a **VM snapshot before the Friday window** and
|
||
**auto-expire it after ~1 week** if health checks stayed green.
|
||
|
||
### 5. Stateful upgrades — 8-weekly analysis, human-gated, backup-first
|
||
|
||
Stateful services are **never** touched by the weekly run. Instead, **every 8 weeks**
|
||
an automated analysis job (a scheduled `claude -p`, per the `scheduled_jobs` design in
|
||
`docs/TODO.md` 8.3, not yet built) does:
|
||
|
||
1. Read changelogs / breaking-change notes for each pinned stateful image; diff the
|
||
pinned tag against what's available.
|
||
2. Emit a **recommended upgrade plan** as a Forgejo issue/PR — proposed target version,
|
||
migration steps, and a backup-first checklist — for a human to approve.
|
||
3. On approval, the upgrade runs in a **deliberate maintenance window** (not the Friday
|
||
auto-run), and **always dumps the DB / takes a backup first** (ties to the backup
|
||
work — TODO 3.8). DB **major**-version bumps are the highest-risk case and get their
|
||
own migration plan.
|
||
|
||
### 6. The verification gate is the health-check harness
|
||
|
||
"Check everything still works" is the load-bearing 80% of this ADR, and it is the
|
||
**same capability** as the test methodology (ADR-008) and the sanity checks already on
|
||
the TODO (2.2 — API/curl/log/headless checks; 8.2 — "does PhotoPrism have its
|
||
pictures?", "is email flowing?"). The update pipeline is a scheduler wrapped around
|
||
that harness.
|
||
|
||
**Sequencing is deliberate: the health-check harness is built first, and no update
|
||
automation (decisions 3–5) is deployed until it is in order.** This is not a blocker to
|
||
work around — it's the order of operations. An update run without a working verification
|
||
gate is "update and pray," so it simply does not ship until the gate is real.
|
||
|
||
### 7. Security fast-path overrides the slow cadence
|
||
|
||
The 8-weekly stateful cadence is for routine drift. It is **too slow for a critical CVE**
|
||
in an internet-facing stateful service (Vaultwarden, Nextcloud, Forgejo). DIUN's
|
||
new-image alert stays as the **out-of-band trigger**: an urgent advisory gets a manual,
|
||
backup-first upgrade immediately, not in up to 8 weeks. Routine = scheduled; urgent =
|
||
alert-driven.
|
||
|
||
---
|
||
|
||
## Open questions (for discussion)
|
||
|
||
1. **Where does the Proxmox snapshot get driven from?** Control node calling the Proxmox
|
||
API as a pre-step in the run, vs a Proxmox-side hook. Crosses the Terraform/Ansible
|
||
boundary (ADR-006/009) — TF "owns VM existence," but a snapshot isn't existence.
|
||
2. **Exact cadences** — Friday weekly and 8-weekly stateful are starting points. Is
|
||
weekly OS patching the right rhythm, or should reboots be rarer than `apt` upgrades?
|
||
3. **Where does the health-check harness live, and what is the minimum bar that counts
|
||
as "in order"** before the weekly run ships (decision 6 fixes the sequencing; this
|
||
pins down the threshold)?
|
||
4. **Classification home** — a per-role `__stateful` flag (proposed) vs a list in
|
||
group_vars.
|
||
5. **Staging first?** Should the weekly run hit a staging host before production, or is
|
||
snapshot-before + Friday timing enough for a homelab of this size?
|
||
6. **Notification + control channel** — boma defines its own ntfy topics (decided
|
||
fresh per ADR-013, not reused from V4), and does the run need a "skip this week" /
|
||
"pause updates" switch? (Relates to TODO item 9 — a tool→user messaging function.)
|
||
|
||
---
|
||
|
||
## What was ruled out
|
||
|
||
| Option | Reason |
|
||
| -------------------------------------- | ----------------------------------------------------------------------------- |
|
||
| One uniform policy for all services | Ignores blast radius; stateful data loss ≠ stateless re-pull. |
|
||
| Rolling `latest` for stateful services | Unattended schema/migration changes are how you lose data. |
|
||
| Digest-_only_ pin (no readable tag) for stateful | Unreadable in diffs — the tiered rule pins `tag@digest` (readable tag *and* digest) instead (Decision 2). |
|
||
| Pinning the stateless tier | No durable data to protect; pins just add churn DIUN already covers. |
|
||
| Auto-updating stateful on a timer | Must be human-gated and backup-first; only the _analysis_ is automated. |
|
||
| Updating the whole fleet at once | Simultaneous reboots hide which host/phase actually broke. |
|
||
| 8-weekly as the only stateful path | Too slow for urgent CVEs — hence the DIUN security fast-path. |
|
||
|
||
---
|
||
|
||
## Consequences
|
||
|
||
- A single uniform update policy is rejected: the stateful/stateless split is
|
||
load-bearing, so stateless services roll on rolling tags while stateful services are
|
||
pinned `tag@digest`, human-gated, and backup-first (see "What was ruled out").
|
||
- The weekly run never touches stateful services and the whole fleet is never updated
|
||
at once, accepting the added orchestration of host ordering and an 8-weekly +
|
||
fast-path cadence in exchange for bounded blast radius (see "What was ruled out").
|
||
- No update automation ships until the health-check verification gate is in order; the
|
||
pipeline is deliberately sequenced behind that harness (see Decision 6).
|
||
- Several points remain open for discussion (see "Open questions"): where the Proxmox
|
||
snapshot is driven from across the TF/Ansible boundary; the exact cadences; where the
|
||
health-check harness lives and the minimum bar that counts as "in order"; whether
|
||
classification is a per-role `__stateful` flag or a group_vars list; whether the
|
||
weekly run hits staging first; and the notification + "skip/pause" control channel.
|