Runbooks¶
Operational procedures for the Internal Developer Platform. Each runbook maps to a Prometheus alert and provides step-by-step remediation guidance.
First-time setup issues? See docs/TROUBLESHOOTING.md — it covers fresh-clone failures like empty catalog, ArgoCD no apps, Backstage port conflicts, and the KAgent HSTS browser issue.
Alert → Runbook Map¶
| Alert | Severity | Group | Runbook |
|---|---|---|---|
PodCrashLooping |
Critical | kubernetes |
Pod Crash Loop |
HighMemoryUsage |
Warning / Critical | kubernetes |
High Memory |
HighCPUUsage |
Warning | kubernetes |
High CPU |
DeploymentReplicasMismatch |
Warning | kubernetes |
Deployment Rollback |
HighHTTP5xxRate |
Critical | kubernetes |
Deployment Rollback |
| RDS unavailable | Critical | database |
Database Recovery |
ImagePullBackOff (local) |
Warning | kubernetes |
ImagePullBackOff |
Kind node NotReady after crash (local) |
Warning | local |
Kind Node IP Mismatch |
TeamBudgetWarning |
Warning | finops |
Cost Budget Exceeded |
TeamBudgetExceeded |
Critical | finops |
Cost Budget Exceeded |
ScaffoldServiceHighRate |
Warning | ai-ops |
KAgent Guardrails |
McpToolErrorRateHigh |
Warning | ai-ops |
KAgent Guardrails |
Severity vocabulary: every Critical/Warning above maps onto P1/P2/P3 — see
Incident Severity for the mapping and which severities get a
tracked incident record.
| SLOErrorBudgetFastBurn | Critical | slo | Deployment Rollback |
| SLOErrorBudgetSlowBurn | Warning | slo | Deployment Rollback |
| Regional outage (multi-region V2 only) | Critical | dr | DR Region Failover |
| Crossplane claim rejected by Kyverno | Warning | platform | TROUBLESHOOTING.md § Crossplane claim rejected |
| ExternalSecret error — SecretStore not found | Warning | platform | TROUBLESHOOTING.md § ExternalSecret error |
| team=unknown on DORA metrics | Info | dora | TROUBLESHOOTING.md § team=unknown |
Local Dev Issues¶
These are not alert-driven but are common when running the platform locally with Kind or Rancher Desktop.
| Symptom | Runbook |
|---|---|
Node NotReady + *.idp.local unreachable after Docker/Rancher crash |
Kind Node IP Mismatch |
Pod stuck in ImagePullBackOff |
ImagePullBackOff |
On-Call Escalation¶
- L1 — On-call engineer (Slack alert fires) — follow the runbook, aim to resolve within 30 min
- L2 — Platform team — escalate via Slack
#platform-oncallif unresolved - L3 — AWS Support — for infrastructure-level failures (EKS control plane, RDS)
Incident Process¶
- Acknowledge the Slack alert (add 👀 reaction)
- Open an incident thread in
#incidentswith: service name, start time, symptoms - Follow the relevant runbook
- Post resolution summary + timeline to the incident thread
- File a post-mortem within 48 hours for P1/P2 incidents — use the post-mortem template