# Design — Logging and log integrity (ship all logs to Loki) - **Date:** 2026-06-05 - **Status:** Approved design — pending implementation plan - **Resolves:** TODO 3.1 ("Decide how to manage logs"); makes concrete ADR-002's "logs shipped to a central location" + "active alerting" controls; advances TODO 3.6 - **Becomes:** ADR-018 (this design is the basis for that ADR) --- ## Problem boma wants **all logs in one queryable store** for three things: day-to-day troubleshooting, spotting issues/trends over time, and **detecting intrusions / malicious activity**. ADR-002 already commits in principle ("`auditd`… Logs shipped to a central location if a log aggregation service is available"; "Active alerting wires AIDE/`auditd`/`fail2ban`/Suricata into the monitoring/alerting stack… ties to the Loki/Grafana effort"), and CAPABILITIES lists Loki (planned) + `askari` as the off-site watchdog. What's undecided is the **architecture** and, critically, the **integrity** dimension: an attacker who roots a host will try to clear logs to cover their tracks. The key insight that frames the integrity question: **the biggest anti-tampering win is that logs leave the host in near-real-time.** Once a line is in a store the attacker doesn't control, wiping the local copy is futile. The remaining question is only *how far* to harden the central store — set by the threat model. ## Decisions (the settled forks) 1. **Threat model — opportunistic + blast-radius**, per ADR-002 / accepted-risk R1. Not forensic-grade. This sizes everything below. 2. **Ship all logs to an on-cluster Loki** — the single monitoring DB for troubleshooting + trends. Near-real-time shipping already defeats per-host track-covering. 3. **Split: a security-relevant subset ALSO ships off-site to `askari`, write-only.** Tamper-resistant against full-cluster compromise, at bounded volume. 4. **Skip WORM/object-lock (Tier 3)** — recorded as accepted-risk R4; append-only push + off-site is the proportionate control. 5. **Disk-wear is a managed design parameter, not a blocker** — storage media choice + bounded verbosity + tuned retention + wearout monitoring (Section: Retention & wear). ## Architecture & components **Agent — Grafana Alloy on every host, installed by the `base` role.** Alloy reads journald + container logs + the security sources (`auditd`, `authpriv`, `fail2ban`, AIDE) on every host (docker_hosts, proxmox nodes, `ubongo`, `askari`) and ships them. Placing it in `base` ties it to ADR-002's baseline "logs shipped to central" control. **Two Loki instances, one Grafana:** ``` ┌──────────────────── per host (base role) ─────────────────────┐ │ Grafana Alloy: collect journald + container + auditd/auth/... │ └──────────┬───────────────────────────────────┬────────────────┘ ALL logs │ security subset │ (over the NetBird mesh) ▼ ▼ ┌────────────────────────┐ ┌──────────────────────────────┐ │ Loki (cluster) all logs│ │ Loki (askari) security only │ │ docker_host, NVMe, │ │ off-site, write-only push, │ │ bounded hot retention │ │ long retention, append-only │ └───────────┬────────────┘ └──────────────┬───────────────┘ └───────────────┬────────────────────┘ ▼ ┌────────────────────────────────────┐ │ Grafana (cluster): both datasources │ │ dashboards + alerts (AIDE/auditd/ │ │ fail2ban/Suricata + log-silence) │ └────────────────────────────────────┘ ``` - **Loki (cluster)** — `loki` service role on a docker_host; **all** logs; monolithic single-binary mode (ample at this scale); NVMe; bounded retention. - **Loki (`askari`)** — the same role parameterised, deployed to the `offsite_hosts` group; **security subset only**, **write-only**, long retention, tiny volume. - **Grafana** — `grafana` service role on the cluster; both Lokis as datasources (one pane queries both); where ADR-002's "active alerting" lands. Reuses what boma already has: `askari` (off-site, on the mesh per ADR-016) and the `base`/service-role machinery. ## Data flow & the security subset Each host's Alloy pipeline writes **everything** to the cluster Loki and a **filtered copy** of security events to the `askari` Loki — a relabel/match stage tags security sources (`security="true"`) and routes only those to the second `loki.write` target. One agent, two destinations. **Security subset** (high-value, bounded volume): `auditd` (auth, privilege, file watches), `authpriv` (SSH, `sudo`), `fail2ban` (bans), AIDE (file-integrity reports), **Suricata** (OPNsense isn't a `base` host, so it **syslog-forwards** alerts to the ingest point), and key container security events (reverse-proxy 401/403, Authentik login events, Docker daemon events). **Write-only / append-only** (the tamper-resistance mechanism): - The `askari` Loki push endpoint (`/loki/api/v1/push`) is reachable only over the **NetBird mesh**, with a **push-only credential**; hosts hold *only* that. - Loki's query/admin/delete APIs on `askari` are **not exposed to hosts** (localhost / mesh-ACL'd to operator + Grafana). The push API has no edit/delete verb, so a compromised host can **append but not read/edit/delete**. Deletion needs the admin/compactor API or filesystem — unreachable from a host. - The cluster Loki uses the same push-only credential, blocking per-host log-clearing via API there too. **Reliability:** Alloy buffers (WAL) and retries, so a brief `askari`/mesh outage doesn't lose logs — they flush on reconnect with only a small local buffer. ## Security, integrity & residual risks **Defeated:** opportunistic track-covering (`rm`/`vacuum`) — lines are already off the host; **host pivot to the store** — an attacker rooting any cluster host can append but not delete, and cannot reach `askari`'s admin plane. **The security trail survives full cluster compromise.** **Honest residual risks (conscious, recorded):** 1. **Append-only ≠ cryptographic WORM** — a root-on-`askari` attacker could edit chunk files on disk. Skipping object-lock is **accepted-risk R4**; mitigated by `askari` being minimal/hardened/operator-only/mesh-only. 2. **Un-shipped window** — a few seconds of not-yet-flushed logs live on the host; near-real-time minimises it. Accept. 3. **Agent compromise (forward-looking)** — rooting a host lets the attacker stop *that host's* Alloy or inject *future* false logs, but **cannot alter shipped history**. 4. **Detection as a feature** — a host that **goes silent** (Alloy stops) is an **alert**; the tamper attempt becomes a signal. "Log-source silence" is wired into Grafana alerting. 5. **Credential theft / `askari` outage** — a stolen push credential allows appending noise, not deletion (bounded, rotatable); an `askari` outage buffers on hosts and flushes on reconnect (a very long outage eventually drops oldest — monitor it). **ADR-002 fit:** realises "logs shipped to central" + "active alerting"; the off-site + append-only model is a clean blast-radius-containment enhancement for the opportunistic threat model. ## Retention, sizing & disk-wear **Sizing (estimates — intent-based until measured, like `/capacity-review`):** a 2–5 host homelab generates ~1–3 GB/day raw "typical" (≪1 GB/day quiet; 5–15 GB/day very chatty); Loki compresses ~7–10× → ~0.1–0.4 GB/day stored; the security subset is ~10–20% of that. **Retention (tunable in `group_vars`):** - **Cluster Loki (all logs):** bounded hot retention, start **30–90 days** (~10–35 GB at 90d on NVMe). - **`askari` Loki (security subset):** **1 year+** (~5–25 GB/yr) — small enough to keep the security trail long for over-time detection. - Defaults now; **re-measure real volume after a few weeks live** and tune. **Disk-wear (the lore is real only for specific media/misconfig; mitigated as design rules):** at boma's volume even ~10–40 GB/day of amplified writes is decades of life on a ~600-TBW/TB NVMe. Rules: 1. Log storage on **NVMe/SSD** (or **HDD** for a long-retention cold tier — sequential, endurance-unlimited); **never SD/USB flash**. 2. **Bounded verbosity at source** (sane log levels, selective access logging, a *targeted* `auditd` ruleset) — the one lever that controls wear *and* firehose size. 3. Tuned Loki **retention + compaction** so neither store grows unbounded. 4. **SSD wearout/TBW is a monitored metric** (Proxmox wearout %, `node_exporter` smartmon) with an alert — wear is a graph, not a surprise. (Depends on the metrics stack — see Dependencies.) Capacity bookkeeping ties into ADR-012: a log-storage allocation line (cluster + `askari`) and SSD-wearout as a tracked metric. ## Documentation & implementation changes This is a substantial capability → its own ADR-018, with reconciliations: | Doc / artifact | Change | |---|---| | ADR-018 (new) | Home of record: ship-all-to-Loki, the off-site write-only security subset, append-only model, skip-WORM (R4), disk-wear rules. | | `base` role (when built) | Install + configure Alloy (all → cluster Loki; subset → `askari` write-only). | | `loki` service role (new, when built) | One role, two deployments (cluster all-logs; `askari` security-subset write-only). `SECURITY.md` + `VERIFY.md`. | | `grafana` service role (new, when built) | Both Lokis as datasources; dashboards + alerting (AIDE/`auditd`/`fail2ban`/Suricata + log-silence). | | OPNsense (Ansible-managed) | Syslog-forward Suricata alerts to the ingest point. | | ADR-002 | "Logs shipped to central" + "active alerting" bullets point to ADR-018. | | `docs/security/accepted-risks.md` | Add **R4** — no cryptographic WORM for logs (append-only + off-site is the control). | | `docs/CAPABILITIES.md` §3 | Loki → decided; add the off-site security sink + Alloy agent rows; mark the alerting wiring. | | `docs/decisions/012-hardware-capacity.md` | Log-storage allocation (cluster + `askari`) + SSD-wearout tracked metric. | | `STATUS.md` + `docs/TODO.md` (3.1 / 3.6) | Mark "how to manage logs" decided by ADR-018; rows as designed-not-built. | | `vault.yml` | Push-only Loki credential (`vault.loki.*`). | **Buildable now:** ADR-018 + the ADR-002/CAPABILITIES/ADR-012/accepted-risks/STATUS/TODO reconciliations. **Deferred on the stack:** the Alloy-in-`base`, `loki`/`grafana` service roles, OPNsense syslog config, and the live pipeline. ## Dependencies - `base` role + service-role machinery (unbuilt) — STATUS.md. - The running cluster + `askari` (`offsite_hosts`, designed) — ADR-016. - OPNsense automation (for Suricata syslog forwarding) — ADR-007. - The **metrics stack** (Prometheus / `node_exporter`) for SSD-wearout + log-silence alerting — sibling effort, TODO 3.6. ## Deferred / out of scope 1. **WORM / object-lock (Tier 3)** — accepted-risk R4; revisit only if the threat model shifts to targeted/forensic. 2. **The metrics pipeline** (Prometheus/`node_exporter`) — sibling effort; this spec is **logs**. SSD-wearout + silence alerting depend on it. 3. **Cold archival beyond Loki retention** (export to backups) and **structured/parsed per-service log standards** — future refinements. ## What was ruled out | Option | Reason | |---|---| | Everything off-site on `askari` (no on-cluster Loki) | The firehose (tens–hundreds of GB/yr) is disk-hungry on a small VPS; keep volume where storage is cheap (on-cluster) and send only the bounded security subset off-site. | | WORM / object-lock for all logs | Forensic-grade cost for an opportunistic threat model — YAGNI (R4). | | On-cluster-only logging (no off-site copy) | Doesn't survive compromise of the cluster Loki host; the security trail needs to be off-cluster + append-only. | | Volatile (RAM-only) journald to cut writes | Risks losing logs on crash before shipping; persistent-with-size-caps + real-time shipping is safer. | | Promtail / legacy agents | Alloy is the current unified Grafana collector and the V4-aligned choice; one agent for logs (and later metrics). | See also: ADR-002 (security baseline — realised here), ADR-016 (mesh / `askari`), ADR-007 (OPNsense / `askari`), ADR-012 (hardware/capacity), ADR-004 (service-role standard), ADR-011 (health checks — distinct from this).