SRE for sovereign platforms

Sovereign deployments are often on-prem, sometimes air-gapped. That rules out the easy answer — a cloud’s managed monitoring — so reliability has to ride inside the platform itself, on an open stack the operator fully owns.

Telemetry as a control room

Every service emits the three signals through one OpenTelemetry collector into a self-hosted Grafana/Loki/Tempo/Prometheus stack. Logs are structured JSON carrying a trace_id, so a log line, its metric, and its trace are one click apart.

The techniques that matter

• RED + USE. Every endpoint reports Rate, Errors, Duration; every resource reports Utilization, Saturation, Errors. Plus domain metrics that map to outcomes, not vanity.
• Health contracts. /healthz (liveness) and /readyz (readiness, gated on dependencies) let the orchestrator and load balancer make correct decisions.
• SLOs and error budgets. We alert on symptoms users feel — latency, error rate against an SLO — not on every CPU spike. Fewer, truer pages.
• Chaos where it counts. We kill the database mid-transaction, cut the network, and fill the disk in staging. Production will eventually do all three; better to learn the failure modes on our terms.
• Boring backups, tested restores. A backup you’ve never restored is a rumour. DR drills are scheduled, not aspirational.
• Least privilege. Every service account and credential has the minimum scope, justified where it deviates from default.

Why it’s built in, not bolted on

Observability and health endpoints exist from the first commit, not after the first incident. On a platform you hand to a client to run themselves, the 2 a.m. runbook is part of the deliverable — so the dashboards, the alerts, and the “what to do when this fires” notes ship with the code.