boma/docs/testing/gotchas.md

Testing & Molecule gotchas

Durable, point-of-use knowledge for writing and running role tests (ADR-008). Migrated from docs/FRICTION.md by the 2026-06-10 kaizen review. Append here when a testing surprise is worth remembering past the session that hit it.

nftables / nft -c render checks

• nft -c rejects iif "<name>" when the interface is absent — iif resolves to an interface index at load time, so it fails in the Molecule container and would fail identically on any real host before the interface exists (e.g. wt0 before NetBird is up). Use iifname "<name>" (string match, no existence requirement, survives the interface coming and going) for any interface that may be absent.
• The render-and-nft -c (no-apply) Molecule approach earns its keep — it caught the iif/iifname bug deterministically without touching the host kernel. Reuse this pattern (render template → static-check, never apply) for other config-rendering roles.

Molecule (community.docker)

• Molecule's community.docker connection uses ansible_host as the container name (remote_addr). Setting ansible_host as data in a scenario's host_vars (e.g. to give a resolver a fake IP) breaks the connection → UNREACHABLE / "Failed to create temporary directory". Don't override ansible_host in Molecule; feed fixture IPs another way (keep fixtures to zone sources and unit-test IP resolution).

Coverage blind spot: apply-only task paths

• Apply-only task paths have no Level-1 coverage, so safety bugs hide there. Example: an nft auto-rollback snapshot used a bare nft list ruleset (no leading flush ruleset), so the revert was a silent no-op on first apply and errored on later ones — the whole safety net was dead. Molecule never runs the apply (gated off), so only adversarial review + an isolated-netns round-trip test caught it. → For apply/safety paths Molecule can't exercise, validate out-of-band (a throwaway --privileged container with its own netns) and treat a final adversarial review as mandatory, not optional.

Tags on dynamic include_tasks need apply: to reach the included tasks

• A tag on a dynamic include_tasks selects the include statement, not its contents. Tagging include_tasks: x.yml with concern and running --tags concern runs nothing (ok=N changed=0) unless the included tasks are independently tagged. Use include_tasks: {file: x.yml, apply: {tags: [concern]}} to propagate the tag onto the included tasks — mandatory whenever a role uses tags to apply concern-subsets (roles/base/tasks/main.yml and roles/dev_env/tasks/main.yml are the references).
• Molecule converges untagged, so it cannot catch this by default — the bug only shows under make deploy … TAGS=<concern> on a real host (first hit live on askari, M3). See the tag-isolation pattern below to catch it in Molecule instead.
• Check-mode artifact: a service/handler for a not-yet-installed package fails in a first-run --check; guard with when: not ansible_check_mode.

Testing concern-tag isolation in Molecule

• To catch the tag-propagation bug above in Molecule, add a second converge play that applies one concern to a fresh target — include_role with apply: {tags: [config]} — plus a verify assertion that the concern's effect landed. Drive the real partial path with molecule converge -- --tags config.
• Sequence matters: a partial-tag run on a fresh instance fails on cross-concern deps (a config task may need a binary the packages concern installs). The realistic test is full converge → partial --tags re-run (idempotent). Harness pre_tasks (e.g. test-user creation) must be tagged always, or --tags filters them out. (Pattern proven on dev_env, 2026-06-14.)

API / templating roles: render-only tests miss the real call

• For a role whose payload is "render data → external API call" (e.g. public_dns → Gandi LiveDNS), apply=false Molecule + data-only pytest exercise the data file, not the rendered module args — so corrupt-template and API-rejection bugs (item.values resolving to a dict method; Gandi rejecting RFC-7505 null-MX 0 .) sail through both, plus review. Only a real (or --check) call against the API surfaces them.
• → Treat a check-mode run against the real API as a required gate for such roles, or build a render-only assertion that materializes and inspects the rendered module args.

Single-file bind mount + atomic rewrite = stale config (reload-in-place only)

• ansible.builtin.template writes atomically (temp file + rename → a new inode). A Docker single-file bind mount pins the old inode, so a container that reloads config in place (no restart) keeps reading the stale file. Live hit: reverse_proxy bind-mounted the Caddyfile as a single file; caddy reload (in-container) re-read the old inode and silently no-op'd ("config is unchanged"). The new NetBird route never loaded → Caddy never requested its cert → surfaced only as a downstream TLS handshake failure.
• Fix for reload-in-place roles: bind-mount the config directory, not the file (./caddy → /etc/caddy). Directory mounts reflect the inode swap, so the reload sees the new file (proven on askari).
• Restart-based roles are fine with a single-file mount. Sibling case: netbird single-file-mounts config.yaml, but its handler does docker compose restart (not an in-container reload), and a restart re-resolves the bind mount (verified: route count 0 before, 1 after). Rule of thumb: reload-in-place needs a directory mount; restart-based roles don't.