boma/docs/decisions/008-testing.md

ADR-008 — Testing methodology

Context

Ansible roles must be idempotent and correct before they touch production hosts. This document records the testing strategy, what each level covers, and — critically — what is explicitly out of scope for automated testing and why.

---

Three testing levels

Level 1 — Molecule (per role, always required)

Runs in Docker on the control node or in CI. Fast (~5 min per role).

What happens during molecule test:

1. create — start the test container
2. converge — apply the role via converge.yml
3. idempotency — run converge.yml again; fail if any task reports changed
4. verify — assert expected state via verify.yml
5. destroy — remove the container

The idempotency step is non-negotiable. Every role must pass it cleanly.

verify.yml must assert outcomes, not task success:

# Wrong — only proves the task ran
- assert:
    that: result is success

# Right — proves the outcome exists
- ansible.builtin.command: systemctl is-active fail2ban
  changed_when: false
  register: svc
- ansible.builtin.assert:
    that: svc.stdout == "active"

Level 2 — Staging playbook (full stack, real VMs)

make check PLAYBOOK=site followed by make deploy PLAYBOOK=site on Terraform-provisioned staging VMs. Catches inter-role dependencies and ordering issues that Molecule cannot see (e.g., docker_host role requires base to have already run and configured the firewall).

Run before every merge to main.

Level 3 — External smoke test from askari

Once askari is operational: scripted checks from outside the network confirming that public-facing services respond correctly. Catches firewall and reverse proxy configuration issues invisible to Ansible check mode.

---

Molecule test image

No external images. The project builds and hosts its own test image.

Source: .docker/molecule-debian13/Dockerfile Base: debian:trixie-slim (official Debian 13, Docker Hub — only external dependency permitted here, as the base OS image is not substitutable) Registry: forgejo.nyumbani.baobab.band/sjat/molecule-debian13:latest

Build and push with:

make molecule-image        # build locally
make molecule-image-push   # push to Forgejo registry (requires docker login)

The scaffold molecule.yml references this image with pre_build_image: true, meaning Molecule uses the image as-is and does not attempt to build it.

Why not geerlingguy/docker-debian13-ansible? It is a Docker Hub image outside project control. It is not a Galaxy role, but it is an external dependency that can drift, disappear, or introduce unexpected changes. The custom image is functionally equivalent and fully owned.

---

Idempotency requirements

Every role task must satisfy one of these:

Task type	Requirement
apt, template, copy, file, user, group, service	Naturally idempotent — no action needed
command / shell (read-only)	changed_when: false
command / shell (detectable change)	changed_when: result.stdout | length > 0 or equivalent
command / shell (creates a file)	creates: /path/to/artifact
Service restart after config change	Move to a handler; handler fires only when notified
docker compose up -d	Handler only — notified by template change, never runs unconditionally

ansible-lint enforces most of these at lint time. The Molecule idempotency step catches anything lint misses.

---

What Molecule tests — and what it does not

Tested in Molecule

Capability	Notes
Package installation	apt works in the container
File and directory creation, permissions, ownership	Full support
Template rendering and content	Full support
User and group management	Full support
Service installation and systemd enable	Requires the systemd-capable image
Service start/stop	Works for most services in the container
SSH configuration file content	File-level only
fail2ban installation and configuration	Install and config file; not live banning
Docker daemon installation	Works in privileged container
auditd installation and configuration	Install and config file
Idempotency of all of the above	Enforced by Molecule's idempotency step

Not tested in Molecule — explicit exceptions

The following require a real kernel or real hardware and are validated only at Level 2 (staging) or Level 3 (external). This is a conscious, documented decision — not a gap.

Capability	Reason not testable in Molecule
nftables rule loading	Requires nf_tables kernel module; not available in Docker
WireGuard tunnel establishment	Requires wireguard kernel module
unattended-upgrades behaviour	Installs correctly; actual upgrade behaviour requires a real apt environment
DHCP behaviour (OPNsense)	OPNsense is managed by Ansible but not testable in a container
mDNS reflector (Avahi cross-VLAN)	Requires real network interfaces and VLANs
Hardware passthrough (NIC, USB)	Not applicable in containers
Corosync cluster formation	Requires multiple real nodes

For the above, Molecule tests only what it can: that the relevant packages are installed, that configuration files render correctly, and that services are enabled. Behavioural correctness is confirmed on staging.

---

CI pipeline

push to main
  ├── yamllint + ansible-lint          (fast gate, ~1 min)
  └── molecule test (all roles)        (parallel, ~5 min per role)

on green (main)
  ├── review tf-plan if infra changed; make check on staging
  └── [manual approval] make deploy PLAYBOOK=site on staging

promote to production
  └── [manual approval] make deploy PLAYBOOK=site on production

Manual gates are intentional. Automated tests prove correctness in isolation; a human confirms the change is safe to promote.