boma/docs/superpowers/specs/2026-06-05-service-ui-verification-design.md

Design — Service-UI acceptance verification (ADR-008 Level 4)

• Date: 2026-06-05
• Status: Approved design — pending implementation plan
• Resolves: ADR-015 deferred item #2 (browser-E2E verification harness); TODO 2.2 (browser portion) + TODO 2.3 (test users + manual-test instruction)
• Expands: ADR-008 Level 4 (currently a stub)
• Becomes: ADR-017 (this design is the basis for that ADR)

---

Problem

ADR-008 defines testing Levels 1–3 (Molecule, staging deploy, external smoke) and a Level 4 stub: "Claude drives a headless browser from ubongo against a deployed service: loads the rendered UI, creates test users, exercises features, and hands the operator a manual test script." Nothing below Level 4 actually exercises a service's application UI — Molecule tests the role in a container, Level 2 confirms the stack converges, Level 3 confirms public endpoints respond. None answer "does PhotoPrism actually let me log in, upload a photo, and see a thumbnail?" (TODO 8.2).

The operator's original ask: "Claude could spin up a browser and actually see the generated service web-UIs to verify various things. Perhaps even generate test users and test features and instruct me on tests as well." That is TODO 2.2 (headless browsing) + TODO 2.3 (test-user generation + manual-test instruction).

Today Claude "sees" a browser only passively — the /screenshot skill fetches screenshots the operator took on mamba. This harness is the active counterpart: Claude drives the browser itself.

Decisions (the settled forks)

1. Nature — Claude-driven exploratory. Claude navigates the live UI with judgment (look, click, reason about whether it works, notice anything off), not deterministic scripts. This is the distinctive value; a scripted Playwright regression suite is explicitly not built here.
2. Mode — interactive, Claude-in-the-loop. Follows from #1: exploratory judgment can't be a headless cron gate. Scheduled smoke-testing stays out of scope (that is a determinism job for health checks / Uptime Kuma later).
3. Environment — staging, full exercise. Claude creates test users and exercises features (including destructive flows) against a staging deploy. Staging is a rebuildable sandbox, so this resolves safety: no production-data risk, no prod pollution.
4. Auth — test users in Authentik (central IdP), real SSO flow. Claude's browser authenticates through Traefik + Authentik exactly as a real user would, faithfully testing the real access path.
5. Structure — per-service VERIFY.md backbone + free exploration. Each service role ships an acceptance spec of critical user journeys; Claude executes it and explores beyond it. Repeatable + intent-capturing, without losing exploratory value.

Scope

In scope: the browser/UI verification harness (TODO 2.2 browser portion) + the test-user and manual-test-instruction standards (TODO 2.3) = ADR-008 Level 4.

Out of scope (siblings, noted not built): the other TODO-2.2 "live testing" methods — API calls, curl pulls, log review. They share the spirit but are not browser work. Also out: a scripted/CI regression suite; scheduled headless smoke checks.

---

Architecture, mechanism, and workflow placement

Mechanism. Claude drives a real Chromium on ubongo via the playwright Claude Code plugin (already earmarked in claude-code-setup.md, enabled when this lands). No bespoke browser code — Claude calls the Playwright MCP tools (navigate, click, type, screenshot, read DOM) and reasons over what it sees. Active counterpart to the passive /screenshot-from-mamba pattern.

Orchestration. A boma skill/command — /verify-service <name> — run interactively on ubongo. It:

1. Reads the service's roles/<name>/VERIFY.md acceptance spec.
2. Provisions/uses a test user in the staging Authentik.
3. Drives the browser through the real SSO flow into the staging service.
4. Executes the listed journeys exploratorily (judging pass/fail, screenshotting key states) and free-explores.
5. Writes a dated verification report with linked screenshots.
6. Emits a manual-test checklist for anything it couldn't do.

Pipeline placement. Level 4 runs after Level 2 (staging deploy) and before production promotion: build role → molecule (L1) → staging deploy (L2) → /verify-service (L4) → promote. It reaches the staging service over the LAN from ubongo (services on srv; resolved via boma DNS), through Traefik + Authentik as a real user would.

Boundaries (one unit, clear interface): the skill orchestrates; VERIFY.md declares intent (per service); Authentik provides identity; the report captures results. Each is independently understandable and swappable.

---

The VERIFY.md standard

Every service role ships a populated roles/<service>/VERIFY.md, copied from a new template docs/testing/service-verify-template.md — parallel to how each role ships SECURITY.md from service-security-template.md. It becomes a role convention (every service role must have a populated VERIFY.md).

Contents:

• Critical user journeys — the acceptance criteria that define "working" for this service (e.g. PhotoPrism: SSO login → library loads → upload a test image → thumbnail generates → search finds it).
• What good looks like — states/screenshots to confirm.
• Not browser-verifiable — items to route to the manual-test handoff (hardware, paid/external flows, subjective quality).

/verify-service reads roles/<name>/VERIFY.md, executes those journeys, and explores beyond them.

Test-user generation standard (TODO 2.3)

Test identities are provisioned in the staging Authentik (never the production IdP — test accounts must not exist in prod):

• Convention: a dedicated test group / naming prefix (e.g. test-<service>@…) so accounts are identifiable and bulk-removable.
• Credentials: ephemeral, generated per run (staging is rebuildable); held only for the run. No test creds in vault.yml.
• Idempotent: reuse-or-create.
• Teardown: primary teardown is the staging rebuild (sandbox); the skill also offers explicit cleanup of the test group.

Reporting & manual-test handoff

• Report: /verify-service writes docs/testing/reviews/YYYY-MM-DD-<service>.md (plus latest.md), mirroring /review-repo→docs/reviews/ and /capacity-review→docs/hardware/reviews/. It contains pass/fail per VERIFY.md journey, observations, the test-user/env used, a verdict, and the manual-test checklist. The committed markdown is the durable artifact.
• Screenshots: saved to a git-ignored dir on ubongo (PNGs would bloat the repo); the report links them and inlines only a few key evidence shots.
• Manual-test handoff (TODO 2.3): anything Claude can't do — physical device, paid/external flow, subjective judgment — becomes a structured checklist in the report (numbered steps, expected result, why handed off). The operator runs them and reports back. This is the "instruct me on tests" half of the vision, as a first-class output.

Safety

Even though staging is a sandbox:

• Staging-only guard. The skill refuses to run against production (verifies it is pointed at the staging environment/inventory before acting) — an ADR-002-aligned hard stop, since exploratory clicking is destructive by nature.
• Confined blast radius. Test users live only in the staging test group; the run sticks to the target service.
• No secrets leaked. Screenshots can capture on-screen tokens/credentials, so the git-ignored screenshot dir is also the safety boundary (evidence isn't committed by default), and the skill avoids capturing credential screens.

---

Documentation & implementation changes

This is a substantial capability → its own ADR-017, with reconciliations:

Doc / artifact	Change
ADR-017 (new)	Home of record: harness, the five settled forks, VERIFY.md standard, test-user + manual-handoff standards, safety.
ADR-008 (testing)	Expand the Level 4 stub into the full definition; link ADR-017.
docs/testing/service-verify-template.md (new)	The VERIFY.md template (parallels service-security-template.md).
.claude/commands/verify-service.md (new)	The /verify-service <name> orchestrating skill.
CLAUDE.md	Role conventions: every service role must ship a populated VERIFY.md. Further reading: ADR-017.
docs/security/service-checklist.md	Add "passed Level 4 (/verify-service)" to the pre-production service-clearance gate.
.gitignore + docs/testing/reviews/	Ignore the screenshot dir; create the reviews dir (README/.gitkeep).
STATUS.md	Row: Level 4 verification — skill + template authorable; running deferred.
docs/TODO.md	Mark 2.2 (browser portion) + 2.3 addressed by ADR-017; note API/curl/log siblings remain.
make new-role scaffold	Scaffold VERIFY.md into new service roles (when that scaffold is next touched).

Buildable now (no ubongo/Authentik/staging needed): ADR-017, the ADR-008 expansion, the VERIFY.md template, the /verify-service skill logic, the convention + checklist + Further-reading edits, .gitignore/dir, STATUS/TODO. This spec yields real working artifacts immediately — the skill and standards exist and are reviewable; only the live run waits on the stack.

Deferred (needs the stack): actually running it (ubongo + playwright plugin + Authentik + a staging deploy); the Authentik test-user provisioning automation; per-service VERIFY.md files (need the service roles, which don't exist yet).

---

Dependencies

• ubongo (ADR-015) — the host that runs the browser. Designed, not built.
• playwright Claude Code plugin — enabled when this lands (claude-code-setup.md).
• Authentik (CAPABILITIES §2, planned) — central IdP for test users + SSO.
• A staging environment with the service deployed (ADR-008 Level 2) — staging is currently empty stubs.

---

What was ruled out

Option	Reason
Scripted Playwright regression suite	The operator wants exploratory judgment, not deterministic scripts; scripts add authoring/maintenance burden. A scripted layer could come later but is not this.
Scheduled headless smoke gate (cron)	Needs determinism, which the exploratory nature excludes; that role belongs to health checks / Uptime Kuma.
Verify against production	Exploratory clicking + test-user creation is destructive/polluting; staging sandbox instead. Production gets non-destructive checks elsewhere, not here.
Free-form exploration with no per-service spec	Flexible but non-repeatable and can miss a service's critical flow; VERIFY.md gives a backbone while keeping free exploration.
Staging bypasses SSO / per-app local users	Wouldn't exercise the real Traefik+Authentik access path; central test users in Authentik are faithful.
Commit screenshots to the repo	Repo bloat + secret-leak risk; git-ignored on ubongo, markdown report committed.

See also: ADR-008 (testing — expanded), ADR-015 (control host — runs the browser), ADR-002 (security), ADR-004 (one service = one role — VERIFY.md parallels SECURITY.md), ADR-013/014 (heritage / knowledge sourcing).