boma/docs/superpowers/specs/2026-06-05-logging-log-integrity-design.md

# Design — Logging and log integrity (ship all logs to Loki)

- **Date:** 2026-06-05
- **Status:** Approved design — pending implementation plan
- **Resolves:** TODO 3.1 ("Decide how to manage logs"); makes concrete ADR-002's
  "logs shipped to a central location" + "active alerting" controls; advances TODO 3.6
- **Becomes:** ADR-018 (this design is the basis for that ADR)

---

## Problem

boma wants **all logs in one queryable store** for three things: day-to-day
troubleshooting, spotting issues/trends over time, and **detecting intrusions /
malicious activity**. ADR-002 already commits in principle ("`auditd`… Logs shipped to
a central location if a log aggregation service is available"; "Active alerting wires
AIDE/`auditd`/`fail2ban`/Suricata into the monitoring/alerting stack… ties to the
Loki/Grafana effort"), and CAPABILITIES lists Loki (planned) + `askari` as the off-site
watchdog. What's undecided is the **architecture** and, critically, the **integrity**
dimension: an attacker who roots a host will try to clear logs to cover their tracks.

The key insight that frames the integrity question: **the biggest anti-tampering win is
that logs leave the host in near-real-time.** Once a line is in a store the attacker
doesn't control, wiping the local copy is futile. The remaining question is only *how
far* to harden the central store — set by the threat model.

## Decisions (the settled forks)

1. **Threat model — opportunistic + blast-radius**, per ADR-002 / accepted-risk R1.
   Not forensic-grade. This sizes everything below.
2. **Ship all logs to an on-cluster Loki** — the single monitoring DB for
   troubleshooting + trends. Near-real-time shipping already defeats per-host
   track-covering.
3. **Split: a security-relevant subset ALSO ships off-site to `askari`, write-only.**
   Tamper-resistant against full-cluster compromise, at bounded volume.
4. **Skip WORM/object-lock (Tier 3)** — recorded as accepted-risk R4; append-only push
   + off-site is the proportionate control.
5. **Disk-wear is a managed design parameter, not a blocker** — storage media choice +
   bounded verbosity + tuned retention + wearout monitoring (Section: Retention & wear).

## Architecture & components

**Agent — Grafana Alloy on every host, installed by the `base` role.** Alloy reads
journald + container logs + the security sources (`auditd`, `authpriv`, `fail2ban`,
AIDE) on every host (docker_hosts, proxmox nodes, `ubongo`, `askari`) and ships them.
Placing it in `base` ties it to ADR-002's baseline "logs shipped to central" control.

**Two Loki instances, one Grafana:**

```
        ┌──────────────────── per host (base role) ─────────────────────┐
        │ Grafana Alloy: collect journald + container + auditd/auth/...  │
        └──────────┬───────────────────────────────────┬────────────────┘
      ALL logs     │                  security subset    │ (over the NetBird mesh)
                   ▼                                     ▼
        ┌────────────────────────┐        ┌──────────────────────────────┐
        │ Loki (cluster) all logs│        │ Loki (askari) security only  │
        │ docker_host, NVMe,     │        │ off-site, write-only push,   │
        │ bounded hot retention  │        │ long retention, append-only  │
        └───────────┬────────────┘        └──────────────┬───────────────┘
                    └───────────────┬────────────────────┘
                                    ▼
                    ┌────────────────────────────────────┐
                    │ Grafana (cluster): both datasources │
                    │ dashboards + alerts (AIDE/auditd/   │
                    │ fail2ban/Suricata + log-silence)    │
                    └────────────────────────────────────┘
```

- **Loki (cluster)** — `loki` service role on a docker_host; **all** logs; monolithic
  single-binary mode (ample at this scale); NVMe; bounded retention.
- **Loki (`askari`)** — the same role parameterised, deployed to the `offsite_hosts`
  group; **security subset only**, **write-only**, long retention, tiny volume.
- **Grafana** — `grafana` service role on the cluster; both Lokis as datasources (one
  pane queries both); where ADR-002's "active alerting" lands.

Reuses what boma already has: `askari` (off-site, on the mesh per ADR-016) and the
`base`/service-role machinery.

## Data flow & the security subset

Each host's Alloy pipeline writes **everything** to the cluster Loki and a **filtered
copy** of security events to the `askari` Loki — a relabel/match stage tags security
sources (`security="true"`) and routes only those to the second `loki.write` target.
One agent, two destinations.

**Security subset** (high-value, bounded volume): `auditd` (auth, privilege, file
watches), `authpriv` (SSH, `sudo`), `fail2ban` (bans), AIDE (file-integrity reports),
**Suricata** (OPNsense isn't a `base` host, so it **syslog-forwards** alerts to the
ingest point), and key container security events (reverse-proxy 401/403, Authentik
login events, Docker daemon events).

**Write-only / append-only** (the tamper-resistance mechanism):
- The `askari` Loki push endpoint (`/loki/api/v1/push`) is reachable only over the
  **NetBird mesh**, with a **push-only credential**; hosts hold *only* that.
- Loki's query/admin/delete APIs on `askari` are **not exposed to hosts** (localhost /
  mesh-ACL'd to operator + Grafana). The push API has no edit/delete verb, so a
  compromised host can **append but not read/edit/delete**. Deletion needs the
  admin/compactor API or filesystem — unreachable from a host.
- The cluster Loki uses the same push-only credential, blocking per-host log-clearing
  via API there too.

**Reliability:** Alloy buffers (WAL) and retries, so a brief `askari`/mesh outage
doesn't lose logs — they flush on reconnect with only a small local buffer.

## Security, integrity & residual risks

**Defeated:** opportunistic track-covering (`rm`/`vacuum`) — lines are already off the
host; **host pivot to the store** — an attacker rooting any cluster host can append but
not delete, and cannot reach `askari`'s admin plane. **The security trail survives full
cluster compromise.**

**Honest residual risks (conscious, recorded):**
1. **Append-only ≠ cryptographic WORM** — a root-on-`askari` attacker could edit chunk
   files on disk. Skipping object-lock is **accepted-risk R4**; mitigated by `askari`
   being minimal/hardened/operator-only/mesh-only.
2. **Un-shipped window** — a few seconds of not-yet-flushed logs live on the host;
   near-real-time minimises it. Accept.
3. **Agent compromise (forward-looking)** — rooting a host lets the attacker stop *that
   host's* Alloy or inject *future* false logs, but **cannot alter shipped history**.
4. **Detection as a feature** — a host that **goes silent** (Alloy stops) is an
   **alert**; the tamper attempt becomes a signal. "Log-source silence" is wired into
   Grafana alerting.
5. **Credential theft / `askari` outage** — a stolen push credential allows appending
   noise, not deletion (bounded, rotatable); an `askari` outage buffers on hosts and
   flushes on reconnect (a very long outage eventually drops oldest — monitor it).

**ADR-002 fit:** realises "logs shipped to central" + "active alerting"; the off-site +
append-only model is a clean blast-radius-containment enhancement for the opportunistic
threat model.

## Retention, sizing & disk-wear

**Sizing (estimates — intent-based until measured, like `/capacity-review`):** a 2–5
host homelab generates ~1–3 GB/day raw "typical" (≪1 GB/day quiet; 5–15 GB/day very
chatty); Loki compresses ~7–10× → ~0.1–0.4 GB/day stored; the security subset is
~10–20% of that.

**Retention (tunable in `group_vars`):**
- **Cluster Loki (all logs):** bounded hot retention, start **30–90 days** (~10–35 GB
  at 90d on NVMe).
- **`askari` Loki (security subset):** **1 year+** (~5–25 GB/yr) — small enough to keep
  the security trail long for over-time detection.
- Defaults now; **re-measure real volume after a few weeks live** and tune.

**Disk-wear (the lore is real only for specific media/misconfig; mitigated as design
rules):** at boma's volume even ~10–40 GB/day of amplified writes is decades of life on
a ~600-TBW/TB NVMe. Rules:
1. Log storage on **NVMe/SSD** (or **HDD** for a long-retention cold tier — sequential,
   endurance-unlimited); **never SD/USB flash**.
2. **Bounded verbosity at source** (sane log levels, selective access logging, a
   *targeted* `auditd` ruleset) — the one lever that controls wear *and* firehose size.
3. Tuned Loki **retention + compaction** so neither store grows unbounded.
4. **SSD wearout/TBW is a monitored metric** (Proxmox wearout %, `node_exporter`
   smartmon) with an alert — wear is a graph, not a surprise. (Depends on the metrics
   stack — see Dependencies.)

Capacity bookkeeping ties into ADR-012: a log-storage allocation line (cluster +
`askari`) and SSD-wearout as a tracked metric.

## Documentation & implementation changes

This is a substantial capability → its own ADR-018, with reconciliations:

| Doc / artifact | Change |
|---|---|
| ADR-018 (new) | Home of record: ship-all-to-Loki, the off-site write-only security subset, append-only model, skip-WORM (R4), disk-wear rules. |
| `base` role (when built) | Install + configure Alloy (all → cluster Loki; subset → `askari` write-only). |
| `loki` service role (new, when built) | One role, two deployments (cluster all-logs; `askari` security-subset write-only). `SECURITY.md` + `VERIFY.md`. |
| `grafana` service role (new, when built) | Both Lokis as datasources; dashboards + alerting (AIDE/`auditd`/`fail2ban`/Suricata + log-silence). |
| OPNsense (Ansible-managed) | Syslog-forward Suricata alerts to the ingest point. |
| ADR-002 | "Logs shipped to central" + "active alerting" bullets point to ADR-018. |
| `docs/security/accepted-risks.md` | Add **R4** — no cryptographic WORM for logs (append-only + off-site is the control). |
| `docs/CAPABILITIES.md` §3 | Loki → decided; add the off-site security sink + Alloy agent rows; mark the alerting wiring. |
| `docs/decisions/012-hardware-capacity.md` | Log-storage allocation (cluster + `askari`) + SSD-wearout tracked metric. |
| `STATUS.md` + `docs/TODO.md` (3.1 / 3.6) | Mark "how to manage logs" decided by ADR-018; rows as designed-not-built. |
| `vault.yml` | Push-only Loki credential (`vault.loki.*`). |

**Buildable now:** ADR-018 + the ADR-002/CAPABILITIES/ADR-012/accepted-risks/STATUS/TODO
reconciliations. **Deferred on the stack:** the Alloy-in-`base`, `loki`/`grafana`
service roles, OPNsense syslog config, and the live pipeline.

## Dependencies

- `base` role + service-role machinery (unbuilt) — STATUS.md.
- The running cluster + `askari` (`offsite_hosts`, designed) — ADR-016.
- OPNsense automation (for Suricata syslog forwarding) — ADR-007.
- The **metrics stack** (Prometheus / `node_exporter`) for SSD-wearout + log-silence
  alerting — sibling effort, TODO 3.6.

## Deferred / out of scope

1. **WORM / object-lock (Tier 3)** — accepted-risk R4; revisit only if the threat model
   shifts to targeted/forensic.
2. **The metrics pipeline** (Prometheus/`node_exporter`) — sibling effort; this spec is
   **logs**. SSD-wearout + silence alerting depend on it.
3. **Cold archival beyond Loki retention** (export to backups) and **structured/parsed
   per-service log standards** — future refinements.

## What was ruled out

| Option | Reason |
|---|---|
| Everything off-site on `askari` (no on-cluster Loki) | The firehose (tens–hundreds of GB/yr) is disk-hungry on a small VPS; keep volume where storage is cheap (on-cluster) and send only the bounded security subset off-site. |
| WORM / object-lock for all logs | Forensic-grade cost for an opportunistic threat model — YAGNI (R4). |
| On-cluster-only logging (no off-site copy) | Doesn't survive compromise of the cluster Loki host; the security trail needs to be off-cluster + append-only. |
| Volatile (RAM-only) journald to cut writes | Risks losing logs on crash before shipping; persistent-with-size-caps + real-time shipping is safer. |
| Promtail / legacy agents | Alloy is the current unified Grafana collector and the V4-aligned choice; one agent for logs (and later metrics). |

See also: ADR-002 (security baseline — realised here), ADR-016 (mesh / `askari`),
ADR-007 (OPNsense / `askari`), ADR-012 (hardware/capacity), ADR-004 (service-role
standard), ADR-011 (health checks — distinct from this).