boma/docs/runbooks/rotate-secrets.md

# Runbook — Vault secrets (setup & rotation)

The master vault password lives in **Vaultwarden** (item `boma-ansible-vault`) and is
fetched on demand by `scripts/vault-pass-client.sh` through the `rbw` agent. It is
never stored in a plaintext file on disk. See ADR-002.

---

## One-time — `rbw` setup on a new machine

```bash
# Install rbw (Debian 13: try apt first, else cargo)
sudo apt install rbw || cargo install rbw

# Point it at the homelab Vaultwarden and log in
rbw config set base_url https://vaultwarden.baobab.band
rbw config set email <your-vaultwarden-email>
rbw login            # prompts for your Vaultwarden master password (+ 2FA if enabled)

# Unlock the agent (do this once per terminal session before vault operations)
rbw unlock

# Sanity check — should print the master vault password:
rbw get boma-ansible-vault
```

Once unlocked, `make encrypt/decrypt/check/deploy` and the pre-commit ansible-lint
hook all obtain the password automatically. If the agent is locked you'll see a
clear "run: rbw unlock" error rather than a hang.

---

## Break-glass — vault access during a full cluster outage

The control node `ubongo` (ADR-015) is the tool used to rebuild the cluster, so it
must be able to decrypt the vault even when Vaultwarden (if hosted on the cluster)
is down. `rbw` keeps a **local encrypted copy** of the Vaultwarden vault and decrypts
it **offline** with your Vaultwarden master password — no live server needed for
entries it has already synced. The recovery design therefore requires:

- `rbw` on `ubongo` (and on `mamba`, the break-glass laptop) has **synced at least
  once** while Vaultwarden was reachable (`rbw sync`).
- Your **Vaultwarden master password** is kept **offline** — in a password manager on
  `mamba` and on paper in a safe — independent of any cluster-hosted Vaultwarden.

There is always exactly one irreducible offline root secret; here it is the
Vaultwarden master password. Keep it recoverable without the cluster.

> **To verify (ADR-014, security-relevant):** confirm `rbw` actually decrypts its
> local cache fully offline on your pinned `rbw` version before relying on this.

---

## Rotating a single secret value

1. Ensure the agent is unlocked: `rbw unlock`
2. Decrypt the relevant vault file:
   ```bash
   make decrypt FILE=inventories/production/group_vars/all/vault.yml
   ```
3. Edit the value (keep the nested `vault.<service>.<key>` structure).
4. Re-encrypt:
   ```bash
   make encrypt FILE=inventories/production/group_vars/all/vault.yml
   ```
5. Commit and deploy:
   ```bash
   git add inventories/production/group_vars/all/vault.yml
   git commit -m "Rotate <service> <key>"
   make check PLAYBOOK=site   # verify what will change
   make deploy PLAYBOOK=site
   ```

---

## Rotating the master vault password

This re-keys every encrypted file from the old password to a new one. **Order
matters** — re-key the files *before* updating Vaultwarden, or the fetch script will
hand back the new password while the files are still encrypted with the old one.

1. Unlock the agent: `rbw unlock` (so the script can read the current/old password).
2. Generate a new password and write it to a throwaway file *outside* the repo:
   ```bash
   umask 077; rbw generate --length 40 > /tmp/new_vault_pass
   ```
3. Re-key all vault files (old via the fetch script, new from the temp file):
   ```bash
   find . -path ./.venv -prune -o -name vault.yml -print \
     | xargs ansible-vault rekey \
         --vault-password-file scripts/vault-pass-client.sh \
         --new-vault-password-file /tmp/new_vault_pass
   ```
4. Update the `boma-ansible-vault` item in Vaultwarden to the new password
   (paste the contents of `/tmp/new_vault_pass`), then re-sync: `rbw sync`.
5. Shred the temp file: `shred -u /tmp/new_vault_pass` (or `rm -f`).
6. Verify: `make decrypt FILE=<a vault.yml>` should still work, now via the new
   password. Commit the re-keyed files.

Rotate the master password when someone with access leaves, or if it is suspected
compromised.

---

## Access for additional people / machines

Access is granted in Vaultwarden — share the `boma-ansible-vault` item (or place it
in a shared collection). Each person/machine then runs the one-time `rbw` setup
above. To revoke access, remove the share in Vaultwarden and rotate the master
password (above), since anyone who had it could have copied it.