boma/docs/runbooks/rotate-secrets.md

Runbook — Vault secrets (setup & rotation)

The master vault password lives in Vaultwarden (item boma-ansible-vault) and is fetched on demand by scripts/vault-pass-client.sh through the rbw agent. It is never stored in a plaintext file on disk. See ADR-002.

---

One-time — rbw setup on a new machine

# Install rbw (Debian 13: try apt first, else cargo)
sudo apt install rbw || cargo install rbw

# Point it at the homelab Vaultwarden and log in
rbw config set base_url https://vaultwarden.baobab.band
rbw config set email <your-vaultwarden-email>
rbw login            # prompts for your Vaultwarden master password (+ 2FA if enabled)

# Unlock the agent (do this once per terminal session before vault operations)
rbw unlock

# Sanity check — should print the master vault password:
rbw get boma-ansible-vault

Once unlocked, make encrypt/decrypt/check/deploy and the pre-commit ansible-lint hook all obtain the password automatically. If the agent is locked you'll see a clear "run: rbw unlock" error rather than a hang.

---

Break-glass — vault access during a full cluster outage

The control node ubongo (ADR-015) is the tool used to rebuild the cluster, so it must be able to decrypt the vault even when Vaultwarden (if hosted on the cluster) is down. rbw keeps a local encrypted copy of the Vaultwarden vault and decrypts it offline with your Vaultwarden master password — no live server needed for entries it has already synced. The recovery design therefore requires:

• rbw on ubongo (and on mamba, the break-glass laptop) has synced at least once while Vaultwarden was reachable (rbw sync).
• Your Vaultwarden master password is kept offline — in a password manager on mamba and on paper in a safe — independent of any cluster-hosted Vaultwarden.

There is always exactly one irreducible offline root secret; here it is the Vaultwarden master password. Keep it recoverable without the cluster.

Verified (2026-06-11, ADR-014): confirmed on ubongo with rbw 1.15.0 — with the Vaultwarden host unreachable, rbw sync fails but rbw get boma-ansible-vault still decrypts from the local cache. Re-verify after an rbw major-version bump.

---

Rotating a single secret value

1. Ensure the agent is unlocked: rbw unlock
2. Decrypt the relevant vault file:

   make decrypt FILE=inventories/production/group_vars/all/vault.yml
   

3. Edit the value (keep the nested vault.<service>.<key> structure).
4. Re-encrypt:

   make encrypt FILE=inventories/production/group_vars/all/vault.yml
   

5. Commit and deploy:

   git add inventories/production/group_vars/all/vault.yml
   git commit -m "Rotate <service> <key>"
   make check PLAYBOOK=site   # verify what will change
   make deploy PLAYBOOK=site
   

---

Rotating the master vault password

This re-keys every encrypted file from the old password to a new one. Order matters — re-key the files before updating Vaultwarden, or the fetch script will hand back the new password while the files are still encrypted with the old one.

1. Unlock the agent: rbw unlock (so the script can read the current/old password).
2. Generate a new password and write it to a throwaway file outside the repo:

   umask 077; rbw generate --length 40 > /tmp/new_vault_pass
   

3. Re-key all vault files (old via the fetch script, new from the temp file):

   find . -path ./.venv -prune -o -name vault.yml -print \
     | xargs ansible-vault rekey \
         --vault-password-file scripts/vault-pass-client.sh \
         --new-vault-password-file /tmp/new_vault_pass
   

4. Update the boma-ansible-vault item in Vaultwarden to the new password (paste the contents of /tmp/new_vault_pass), then re-sync: rbw sync.
5. Shred the temp file: shred -u /tmp/new_vault_pass (or rm -f).
6. Verify: make decrypt FILE=<a vault.yml> should still work, now via the new password. Commit the re-keyed files.

Rotate the master password when someone with access leaves, or if it is suspected compromised.

---

Access for additional people / machines

Access is granted in Vaultwarden — share the boma-ansible-vault item (or place it in a shared collection). Each person/machine then runs the one-time rbw setup above. To revoke access, remove the share in Vaultwarden and rotate the master password (above), since anyone who had it could have copied it.