boma/docs/FRICTION.md

# FRICTION.md — kaizen friction log

Raw signals for the periodic **kaizen review** (`/kaizen`; see `docs/TODO.md` 11). This is
the input that keeps our tooling and conventions sharpening over time instead of only
accreting.

**How to use:** append freely _during_ work under **Open signals** — don't curate,
don't fix there. Capture friction, surprises, fixes that keep recurring, and tooling
that isn't earning its keep. `/kaizen` reads this, then proposes a verdict per signal
(SYSTEMATIZE / CHANGE / PARK / REMOVE / ALREADY-BUILT / ACCEPTED / KEEP-OPEN; biased
toward _remove/park_ for unused tooling), migrates durable knowledge into the right docs,
and moves consumed signals into the **decisions ledger** below.

**Entry format:** `date — [tag] observation — (optional) → systematization idea`
Tags: `[friction]` recurring annoyance · `[gotcha]` surprising behaviour ·
`[recurring]` keeps coming back, should be systematized · `[unused]` tooling not
earning its keep.

---

## Open signals

_(append new raw signals here; the next kaizen review consumes them)_

- `[friction]` **Image push to the Forgejo registry fails with `no basic auth
  credentials`** (2026-06-15): `make caddy-image-push` (and `molecule-image-push`) fail
  unless the Docker daemon on ubongo has an interactive `docker login
  forgejo.nyumbani.baobab.band` session — and those creds are **not in vault** (only
  `gandi` + `hetzner` are), so an agent can't complete a push non-interactively. The
  build half is fully automatable; the push half silently requires a human. → candidate:
  document the `docker login` step in `docs/runbooks/claude-code-setup.md`, **or** store
  a scoped Forgejo registry token in vault + a `make registry-login` target (login via
  `--password-stdin`, `no_log`) so pushes are agent-completable like every other
  vault-backed action.

- `[gotcha]` **Single-file Docker bind mount + atomic config rewrite = stale config in
  the running container** (2026-06-16): `reverse_proxy` bind-mounted the Caddyfile as a
  single file; `ansible.builtin.template` writes atomically (temp + rename → new inode),
  so the running container kept the OLD inode and `caddy reload` (in-container, no restart)
  re-read stale config and silently no-op'd (`"config is unchanged"`). The NetBird route
  never loaded → Caddy never requested its cert; surfaced only by a TLS handshake failure.
  Fix: mount the config **directory** (`./caddy` → `/etc/caddy`) — directory mounts reflect
  inode swaps, so live reload works (proven on askari). NOTE the sibling case: NetBird also
  single-file-mounts `config.yaml`, but its handler does `docker compose restart` (not an
  in-container reload), and a restart DOES re-resolve the bind mount (verified: 0 before,
  1 after) — so restart-based roles are safe; only in-place-reload roles need the dir mount.
  → candidate gotcha doc (`docs/testing/gotchas.md`): "reload-in-place needs a directory
  mount; restart-based roles are fine with a single-file mount."

- `[friction]` **`make check` always fails on the first-ever deploy of a compose service
  role** (2026-06-16): in check mode the "ensure base_dir" task is reported-but-not-run, so
  the later `community.docker.docker_compose_v2` up fails with `"…is not a directory"`
  (missing `project_src`). Not a defect — a real deploy creates the dir — but it means the
  CLAUDE.md "always `make check` before `make deploy`" step is guaranteed-red for any brand
  new stateful role, which erodes trust in the check. → candidate: guard the compose-up with
  `not ansible_check_mode` (clean "skipped" in dry-run; compose can't be meaningfully
  dry-run before first deploy anyway), OR document the one-time expected failure. Decide one.

- `[recurring]` **Re-asked the operator about settled defaults — push + execution mode**
  (2026-06-17): at the M5 plan handoff I asked (a) whether to push to origin and (b) which
  execution mode (subagent-driven vs inline) — both already settled: CLAUDE.md says push to
  `origin` often (off-machine backup), and TODO 10.5 / the standing agreement is "always
  subagent-driven" (there's even `guard-execution-mode-menu.sh`). Same shape as the 5×
  "execution-mode menu asked AGAIN" ledger entries — but this time the ask was my own
  free-form prose ("want those pushed now?", "which execution approach?"), which the
  existing menu-text matcher does NOT catch (it keys on the writing-plans menu's literal
  text). → the gap is that the guard only matches that literal menu; free-form re-asks slip
  through. Candidate: widen the Stop-hook matcher to also flag prose re-asks of
  push-vs-not / subagent-vs-inline, since prose reminders have already failed this many
  times. Default behaviour: **push as backup and proceed subagent-driven without asking.**

- `[friction]` **A docs-only commit still tripped the `rbw`-locked pre-commit guard**
  (2026-06-17): committing only `docs/superpowers/specs/*.md` (no ansible content) was
  blocked needing the vault password, although the 2026-06-10 kaizen fix scoped the
  pre-commit `ansible-lint` hook (`always_run: false` + `files:` ansible content) so
  docs-/config-only commits skip it and need no vault. So either the hook's `files:`
  pattern still matches `docs/**` (or `.md`), or a blanket pre-commit step needs the
  vault regardless. → check `.pre-commit-config.yaml`'s `files:`/`exclude:` against the
  spec/plan paths; docs-only commits should not require `rbw`.

- `[friction]` **The agent can't manage `ubongo` (the control node it runs ON) without
  the operator granting access** (2026-06-17): enrolling `ubongo` in the mesh needed two
  manual operator grants because the agent runs as `claude` (no sudo) but the inventory
  manages `ubongo` as `sjat`: (1) `claude`'s SSH key added to `sjat`'s `authorized_keys`
  (`Permission denied (publickey)` otherwise), then (2) `NOPASSWD` sudo for `sjat`
  (`Missing sudo password` otherwise). So the "AI-worker control node" (ADR-015) can drive
  the whole fleet but not itself, unattended. This is the **pending `ansible`-user
  bootstrap** gap (STATUS) biting in practice. → the proper fix is ubongo's bootstrap to a
  key-trusted, NOPASSWD `ansible` (or `sjat`) management identity as part of `base`/its
  control-node recipe, so control-node self-management doesn't need ad-hoc operator grants.

- `[recurring]` **ADRs claim cross-doc reconciliation they didn't actually perform**
  (2026-06-14): ADR-024's Status + Consequences asserted "ADR-017 prose that mentioned
  Traefik is updated to read Caddy" — but ADR-008/017/019 + CAPABILITIES still said
  Traefik; the rename was left half-done across the doc set and the ADR over-claimed its
  own follow-through. Surfaced only by a full-repo `grep Traefik` during `/review-repo`.
  Same shape as the deferred-decision-goes-stale signal (a decision lands in one place,
  its promised ripple edits don't). → candidate `repo-scan.py` check: when an ADR's text
  asserts "X is updated to Y" / supersedes a named tool, flag remaining occurrences of the
  old name (or verify the claimed edit landed) — the structural cousin of `stale-deferred`.
  (KEEP-OPEN per the 2026-06-14 `/kaizen` run — it's its own build task.)

---

## Kaizen reviews — decisions ledger

Consumed signals and where their resolution now lives. Newest first.

### 2026-06-14

First `/kaizen` run (dogfood). 12 signals triaged; 11 consumed, 1 kept open (#13 above —
a `repo-scan.py` check is its own build). **Bias-to-remove note:** zero PARK/REMOVE — none
of the open signals were `[unused]` *tooling*; they were all knowledge/gotchas/process,
which migrate or archive (knowledge is never deleted).

| Signal (first seen) | Verdict | Resolution / where it lives now |
|---|---|---|
| Execution-mode menu asked AGAIN — 5× (06-05→06-14) | ALREADY-BUILT | The 06-10 mechanical guard (`.claude/hooks/guard-execution-mode-menu.sh`, wired in `.claude/settings.json`) is **verified firing** on the real writing-plans menu text (tested 06-14). The 06-14 miss was hook-activation timing (the known "hooks-need-restart" gotcha), not a matcher defect. |
| Brainstorming spec-review gate fires despite the standing agreement (06-10) | CHANGE → mechanical | Extended the same Stop hook with a tight second matcher (review + "the spec" + "before" + "implementation plan", or the literal "spec written and committed"); tested to block the gate and pass meta-discussion. Same external-skill-script-vs-convention family as the execution menu. |
| Subagent faithfulness self-reports can be wrong (06-10) | ACCEPTED | The mitigation — independent two-stage review where the reviewer is told "do not trust the report" and reads the actual diff — is now embodied in `superpowers:subagent-driven-development`, used for the `/kaizen` build itself. Revisit if it recurs. |
| ADR-writing policy unsettled (05-31) | ALREADY-BUILT | ADR-023 (ADR structure & lifecycle) + `docs/decisions/adr-template.md` settle status/sections — both postdate this signal. |
| Hetzner 403 / caddy-dns DNS-01 didn't issue (06-14) | ALREADY-BUILT → **RESOLVED 2026-06-15** | 06-14: ADR-024 recorded the HTTP-01 decision + DNS-01 deferral. 06-15: deferral **closed** — root cause was **version skew** (pre-Bearer `libdns/gandi` sent Gandi's deprecated `Apikey` header → 403) plus building on a Hetzner IP. Fix: pin caddy-dns/gandi v1.1.0 (Bearer PAT) + build on ubongo. DNS-01 now built + proven (real wildcard cert via LE staging). See ADR-024 Status + STATUS.md + `roles/reverse_proxy`. |
| `apply:{tags}` not propagated by dynamic `include_tasks` (06-14) | SYSTEMATIZE | → `docs/testing/gotchas.md` — "Tags on dynamic `include_tasks` need `apply:`". |
| Molecule CAN test tag-propagation, via a tagged converge (06-14) | SYSTEMATIZE | → `docs/testing/gotchas.md` — "Testing concern-tag isolation in Molecule". |
| apply=false Molecule + data-pytest gap for API/templating roles (06-14) | SYSTEMATIZE | → `docs/testing/gotchas.md` — "API / templating roles: render-only tests miss the real call". |
| `item.values` in a loop sends the dict method, not the key (06-14) | SYSTEMATIZE | → CLAUDE.md Ansible conventions ("index loop-var keys with `item['key']`, never `item.key`"). |
| TF child modules need their own `required_providers` (06-14) | SYSTEMATIZE | → CLAUDE.md Terraform conventions ("every module declares its own `required_providers` in `versions.tf`"). |
| ansible-lint `var-naming` rejects `access__`/`backup__` cross-role names (06-14) | SYSTEMATIZE | → `make new-role` scaffolds a noqa reminder in `defaults/main.yml`; ADR-004's service-role section documents the convention; `roles/reverse_proxy/defaults/main.yml` is the reference. |
| Gandi rejects RFC-7505 null-MX `0 .` (06-14) | MIGRATE | → `roles/public_dns/README.md` Notes (no MX + SPF `-all` + DMARC reject for a no-mail domain). |

### 2026-06-10

| Signal (first seen) | Verdict | Resolution / where it lives now |
|---|---|---|
| Execution-mode menu asked at plan handoff — 4× (06-05/06/09/10) | CHANGE → mechanical | Stop hook in `.claude/settings.json` blocks the turn if the menu appears and tells me to proceed subagent-driven. Prose reminders (CLAUDE.md, memory, 3 FRICTION entries) had failed four times — the lesson is that a behaviour conflicting with an external skill's script needs a *mechanical* guard, not another note. |
| Every `git commit` needs `rbw` unlock — recurring (05-30) | CHANGE | Root cause was **not** the vault syntax-check (`.ansible-lint` already excludes `vault.yml`); it was ansible-lint auto-loading + decrypting `inventories/production/group_vars/all/vault.yml` via the wired `vault_password_file`. Scoped the pre-commit `ansible-lint` hook (`always_run: false` + `files:` ansible content) so **docs-/config-only commits skip it and need no vault**. Ansible-content commits still need `rbw` (intrinsic to linting vault-backed plays; accepted). |
| `make test` fails when run non-activated — `ansible-config` not found (06-06) | CHANGE | `Makefile` `test`/`test-all` now prepend `$(CURDIR)/.venv/bin` to `PATH`. |
| Molecule image missing from the Forgejo registry (06-06) | already built | `make molecule-image-push` target exists. |
| Deferred decision goes stale across docs — 3× (06-05) | already built | `scripts/repo-scan.py` `open-deferred-item` / `stale-deferred` checks, run by `/review-repo`. |
| `make new-role` brace-expansion fails under dash (05-30) | fixed | Explicit paths in the Makefile target. |
| nft `iif` vs `iifname`, Molecule `ansible_host`, apply-path coverage blind spot, render-`nft -c` pattern (06-06) | MIGRATE | → `docs/testing/gotchas.md` (pointer from ADR-008). |
| hooks-need-restart, pre-commit stashes unstaged, `rbw sync` stale cache, zsh word-split (05-30) | MIGRATE | → `docs/runbooks/claude-code-setup.md` "Environment gotchas". |
| `finishing-a-development-branch` offers open-a-PR vs our trunk-based merge (06-01) | accepted | Same root cause as the menu ask (external skill script vs boma convention). CLAUDE.md already mandates trunk-based merge-to-main; covered by the Stop-hook family + awareness. Revisit if it recurs. |

**Process note:** the 2026-06-10 review was manual (the `/retro`/`/kaizen` tool wasn't
built). The 2026-06-14 block was the **first run of `/kaizen`** itself
(`scripts/friction-scan.py` Phase 0 + `.claude/commands/kaizen.md`); the dogfood both
cleared the backlog and validated the command.