SRE & Reliability¶

This page covers the reliability practices and tooling shipped as part of the platform's SRE programme (Sprints 1–7). All items described here are already deployed and active.

SLOs and Error Budgets¶

Overview¶

Service Level Objectives are defined using Sloth, a YAML-based SLO generator that produces multi-window, multi-burn-rate Prometheus alerting rules from a simple specification. Sloth SLOs ship as the platform default for all new services via the slo-definition Backstage template.

Defining an SLO¶

Use the Backstage scaffolder to add an SLO to any service:

Go to Create → SLO Definition
Fill in service name, target (e.g. 99.5), and latency threshold
The template writes a Sloth YAML to observability/slo/<service>-slos.yaml and opens a PR

Example reference: observability/slo/hello-service-slos.yaml — 99.5% availability, p99 < 500 ms.

Platform SLOs that ship with the repo:

Source	SLOs	Alerts
`observability/slo/hello-service-slos.yaml`	99.5% availability, 99% of requests < 500 ms	page + ticket
`observability/slo/mcp-servers-slos.yaml`	99% tool-call availability for each of the 8 MCP servers (`mcp_tool_calls_total{outcome="error"}`)	ticket only: tool-call volume is too low and bursty to page on

bootstrap.sh (AWS) and bootstrap-local.sh (Kind / Rancher Desktop) both apply every file in observability/slo/generated/.

How SLOs reach the cluster¶

The PrometheusServiceLevel in observability/slo/ is a Sloth CRD, and no Sloth operator runs in either environment, so it has to be compiled into a PrometheusRule first. That happens ahead of time: ./scripts/generate-slo-rules.sh compiles every source with a pinned sloth build into observability/slo/generated/, and the bootstraps apply that directory. No local sloth install is needed.

If generated/ is empty the bootstrap warns and the Grafana SRE dashboard reports "no sloth_slo_info metrics found" — which is the symptom to look for when error budgets are empty on a fresh cluster. After editing an SLO source file, run ./scripts/generate-slo-rules.sh and commit observability/slo/generated/, or the cluster keeps applying the stale committed copy. The script downloads a pinned, checksum-verified sloth build, so it needs no local install, and the slo-drift CI job runs it with --check to fail any PR where a source and its generated rules disagree (or a new source has no generated file).

Multi-window burn-rate alerts¶

The platform ships two alert groups in observability/alertmanager/prometheus-rules.yaml:

Group	Alert	Window	Burn rate	Action
`slo-burn-rate`	`SLOErrorBudgetFastBurn`	1 h / 5 min	> 14×	Page on-call (Critical)
`slo-burn-rate`	`SLOErrorBudgetSlowBurn`	6 h / 30 min	> 2×	Slack warning
`dora-anomalies`	`HighChangeFailureRate`	24 h	CFR > 10%	Slack warning

Viewing SLO status in Grafana¶

http://grafana.idp.local/d/slo

The SLO dashboard shows current error budget burn rate, remaining budget percentage, and a 30-day trend per service.

Backstage Metrics¶

The Backstage backend exposes Prometheus metrics on :9464/metrics (backstage/app/packages/backend/src/instrumentation.js, preloaded with node --require ./instrumentation.js by every start command). It provides:

http_server_request_duration_seconds_* from the OpenTelemetry HTTP instrumentation, labelled by method and http_response_status_code. Health probes (/ready, /healthz, /healthcheck) are excluded, so these counts are real traffic: the basis for a Backstage availability or latency SLO.
Backstage's own metrics (catalog processing, scaffolder and scheduler tasks), which it records through @opentelemetry/api and which had nowhere to go without a MeterProvider.

Environment	How it is scraped
AWS	`aws/backstage/metrics.yaml`: a ClusterIP Service plus a ServiceMonitor (`release: prometheus`), applied by `bootstrap.sh`. Deliberately not on the public NLB Service.
Local	Published on `127.0.0.1:9464` for `curl`. Not scraped by the in-cluster Prometheus, because Backstage runs in Docker Compose outside Kind.

Turn it off with BACKSTAGE_METRICS_ENABLED=false, or move it with BACKSTAGE_METRICS_PORT.

PodDisruptionBudgets¶

What ships by default¶

The golden-path Helm chart (helm/service-template/) includes a PodDisruptionBudget with maxUnavailable: 1 enabled by default. This stops voluntary disruptions (node drains, cluster upgrades) from evicting more than one replica of a service at a time.

What `maxUnavailable: 1` means in practice¶

With 2+ replicas, a kubectl drain evicts one pod at a time and waits for its replacement, so at least one replica keeps serving.
With a single replica, the pod can still be evicted, so the service is briefly down during a drain. That is deliberate: minAvailable: 1 on one replica allows zero disruptions and blocks node drains and EKS node group upgrades indefinitely (#318). Scale to at least 2 replicas if the service must stay up through a drain.

Overriding¶

# helm-values-local.yaml or helm-values-aws.yaml
podDisruptionBudget:
  enabled: true
  minAvailable: 2   # critical services; takes precedence over maxUnavailable.
                    # Keep it below replicaCount/minReplicas or drains block.

Zone spread¶

A PDB alone does not survive an AZ outage. The chart's pod anti-affinity only keeps replicas on different nodes, and those nodes can share an AZ. Replicas are therefore also spread across zones by default, through a topologySpreadConstraints entry on topology.kubernetes.io/zone:

zoneSpread:
  enabled: true
  maxSkew: 1
  whenUnsatisfiable: ScheduleAnyway   # DoNotSchedule makes it a hard requirement

ScheduleAnyway makes it a scoring preference, so it can never leave a pod Pending. That matters on Kind, whose nodes carry no zone label, and while a node pool is briefly uneven. Use DoNotSchedule only for services whose replicas must never share an AZ, and only with node capacity in every zone.

Blameless Postmortem Process¶

The platform includes a blameless postmortem template at docs/postmortem-template.md.

Process¶

For any P1 or P2 incident, a postmortem must be filed within 48 hours of resolution.
A draft is opened for you. When agent-event-router swaps incident:open for incident:needs-postmortem, the Postmortem workflow (.github/workflows/postmortem.yml) renders docs/postmortems/<INC-id>-<service>.md and opens a PR. The identifiers, the timeline reconstructed from the issue and its comments, and the MTTR arithmetic are filled in; every section marked _TODO_ needs a human.

The root cause is deliberately left blank rather than guessed — a plausible-sounding wrong root cause is worse than an empty heading. A daily job comments on any incident still lacking a draft after 48 hours, so the SLA below is enforced by something other than memory.

To regenerate, re-apply the label, or run the workflow manually with the issue number. 3. Fill in the five sections: timeline, impact, root cause, contributing factors, action items. 4. Open a PR — the platform team reviews within 24 hours. 5. Action items are tracked as GitHub Issues with the postmortem label.

Key principle: The template explicitly focuses on what happened and what we change, not who was at fault.

Incident Record Automation¶

agent-event-router (services/agent-event-router) auto-creates a tracked GitHub issue the moment a Critical-severity alert fires — seeded with the Incident ID, severity, affected service, and start time (mirroring the summary table in docs/postmortem-template.md). When the alert resolves, it posts a comment with the resolution time and duration, swaps the incident:open label for incident:needs-postmortem, and leaves the issue open for the 48-hour postmortem process above. This requires GITHUB_TOKEN and INCIDENT_REPO to be set on the service (see services/agent-event-router/helm-values-local.yaml); without them, the existing AI-triage routing to idp-assistant/cost-agent still works, but no issue is created.

OPA Cost-Tag Enforcement¶

The require-cost-tags OPA/Gatekeeper policy (kubernetes/policies/require-cost-tags.yaml) was upgraded from warn mode to deny mode in Sprint 1. This means:

Any workload (Deployment, StatefulSet, DaemonSet) without idp.io/cost-centre and team labels will be rejected at admission.
The golden-path Helm chart pre-populates both labels from template values so scaffolded services pass automatically.
To add tags to a brownfield workload:

metadata:
  labels:
    team: backend-team
    idp.io/cost-centre: backend

Verify a manifest before applying:

helm template my-svc helm/service-template --values helm-values-local.yaml | \
  kubectl apply --dry-run=server -f -

Log Aggregation — Loki + Promtail¶

Architecture¶

Promtail runs as a DaemonSet and ships all pod stdout/stderr to Loki. Grafana queries Loki via the built-in Loki datasource.

Local (Kind) note: Loki and Promtail ship scaled to 0 and collect nothing until re-enabled — the single-node cluster does not have the capacity to run them alongside the rest of the platform. See local-setup.md → Loki and Tempo ship disabled. AWS is unaffected.

Environment	Loki location
Local (Kind)	`local/observability/loki/` — single-binary mode
AWS (EKS)	`aws/observability/loki/` — scalable distributed mode

Querying logs in Grafana Explore¶

http://grafana.idp.local/explore

Example queries:

# All logs from a service
{app="hello-service"}

# Error logs only
{namespace="services-dev"} |= "error"

# Audit log entries from IDP MCP Server
{app="idp-mcp-server"} |= "[AUDIT]" | json

# Follow logs for a specific pod
{pod="contract-mcp-server-abc123"}

TraceID linking to Tempo¶

Loki is configured with a derived field that detects traceId in log lines and renders it as a clickable link to the matching Tempo trace. This means you can jump from a log line directly to the distributed trace that generated it.

Distributed Tracing — Grafana Tempo¶

Endpoints¶

Endpoint	Protocol	Use
`http://tempo.idp.local`	HTTP	Grafana UI, health checks
`tempo.idp.local:4317`	gRPC (OTLP)	SDK instrumentation
`tempo.idp.local:4318`	HTTP (OTLP)	SDK instrumentation, curl testing

Instrumenting a service¶

Node.js / TypeScript:

import { NodeSDK } from '@opentelemetry/sdk-node';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';

const sdk = new NodeSDK({
  traceExporter: new OTLPTraceExporter({
    url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT || 'http://tempo.idp.local:4318/v1/traces',
  }),
});
sdk.start();

Go:

import "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"

exporter, _ := otlptracehttp.New(ctx,
  otlptracehttp.WithEndpoint("tempo.idp.local:4318"),
  otlptracehttp.WithInsecure(),
)

Set the OTEL_SERVICE_NAME environment variable to your service name so traces appear labelled correctly in Tempo.

Viewing traces¶

http://grafana.idp.local/explore

Select the Tempo datasource and search by service name, trace ID, or span attributes.

Datadog Infra Observability & APM¶

Coexistence with Prometheus/Grafana¶

Datadog and Prometheus/Grafana deliberately own different telemetry domains — this is not a migration off Prometheus:

Domain	Owner
App-level `/metrics`, ServiceMonitors, SLO burn-rate alerting	Prometheus + Grafana + Sloth (see SLOs and Error Budgets)
Cluster/node/pod infra metrics, container logs, APM traces	Datadog Agent

The Datadog Agent does not scrape the Prometheus/OpenMetrics endpoints services already expose (datadog.prometheusScrape.enabled: false in aws/observability/datadog/datadog-agent-values.yaml). This avoids double cardinality billing in Datadog and keeps alert ownership unambiguous — an app-level SLO burn-rate alert always comes from Alertmanager, an infra/host alert always comes from Datadog.

Deployment¶

aws/observability/datadog/datadog-external-secret.yaml — syncs idp-mvp/datadog (Datadog API/App keys) from AWS Secrets Manager into the datadog-secrets K8s Secret via a dedicated IRSA-authenticated ServiceAccount (datadog-eso-sa), same least-privilege pattern as aws/kagent/external-secret.yaml.
aws/observability/datadog/datadog-agent-values.yaml — Helm values for the official datadog/datadog chart: Cluster Agent + node DaemonSet, log collection, APM trace intake on port 8126. Installed directly via helm upgrade --install in bootstrap.sh (Phase 4.4-pre-d), same mechanism as Loki/Tempo above it — not an ArgoCD Application (that pattern is reserved for add-ons that aren't wired into bootstrap.sh).
Site: datadoghq.eu for all Datadog integrations (Agent, Backstage /datadog proxy, dd-trace, scaffolder default) — matches the existing datadog-synthetic-suite template.

APM traces from the Backstage backend¶

The Backstage backend itself is instrumented with dd-trace (NODE_OPTIONS=--require dd-trace/init, see aws/backstage/deployment.yaml), reporting to the Agent DaemonSet on the node via DD_AGENT_HOST=status.hostIP:8126. This is the simplest end-to-end check that the Agent is reachable — look for a backstage service under Datadog APM → Services.

Catalog UI and per-service opt-in¶

Service entity pages show a Datadog card (dashboard link, monitor status, SLO status) when the catalog entity has datadoghq.com/dashboard-url, datadoghq.com/monitor-tag, or datadoghq.com/slo-id annotations. The card calls Datadog through the Backstage backend's /datadog proxy — API/App keys never reach the browser.
Existing services can opt into dd-trace APM + these annotations via the Enable Datadog APM & Monitoring scaffolder template (backstage/catalog/templates/enable-datadog-apm/).

Deployment tracking¶

.github/workflows/build-and-deploy.yml sends a Datadog deployment marker (datadog-ci deployment mark) after each successful smoke test — one in smoke-test-dev (env dev) and one in smoke-test-staging (env staging), tagged with the service name and image tag. This annotates Datadog dashboards/APM traces with exactly when a revision went live, so an incident (a latency spike, an error-rate jump) can be correlated with "which deploy caused it."

Requires the DD_API_KEY repository secret on the platform repo (see docs/DEPLOYMENT_GUIDE.md); the step no-ops if it's unset, and is continue-on-error: true regardless — deployment tracking is observability, not a release gate.
No production marker today: production promotion only opens a PR (promote-to-production) — the workflow has no job that confirms the merge actually deployed (no smoke-test-production exists, unlike dev/staging). Add one if you want a prod-env marker.

Alert Routing¶

All alerts go to #platform-alerts, and critical and warning alerts also go to agent-event-router (observability/alertmanager/alertmanager-config.yaml, in the monitoring namespace).

Per-team channels: set Team Slack alert channel (e.g. #payments-alerts) when running the Team Namespace template, together with the ESO IAM role ARN. The template adds kubernetes/teams/<team>/alertmanager-config.yaml, which routes critical and warning alerts from team-<team> to that channel in addition to #platform-alerts. Store the channel's incoming-webhook URL first:

aws secretsmanager create-secret --name <team>/slack-webhook \
  --secret-string '{"webhook-url":"https://hooks.slack.com/services/..."}'

Until that secret exists, the operator skips only that team's config; platform routing and other teams keep working.

Why the scoping works: the Alertmanager runs with alertmanagerConfigMatcherStrategy: OnNamespaceExceptForAlertmanagerNamespace (prometheus-stack-values.yaml, local and AWS). The config in monitoring sees every alert, and each team's config only sees alerts from its own namespace. Under the operator's default, OnNamespace, the platform config was silently limited to alerts labelled namespace="monitoring", so service alerts and SLO burn alerts never reached Slack or the router (#159).

PagerDuty Escalation¶

Configuration¶

PagerDuty is wired into AlertManager as a receiver for Critical severity alerts. Configure it by setting the integration key in local/.env:

PAGERDUTY_INTEGRATION_KEY=your-events-api-v2-key

The AlertManager config is at observability/alertmanager/alertmanager-config.yaml.

What pages on-call¶

Alert	Trigger condition	Destination
`SLOErrorBudgetFastBurn`	Error budget burns > 14×	PagerDuty (Critical)
`PodCrashLooping`	Pod restart count high	PagerDuty (Critical)
`HighHTTP5xxRate`	5xx rate > 5%	PagerDuty (Critical)
`TeamBudgetExceeded`	Monthly cost > 100% budget	PagerDuty (Critical)

What goes to Slack only¶

Slow-burn SLO alerts, HighMemoryUsage (Warning), TeamBudgetWarning (80%), and ScaffoldServiceHighRate route to Slack #platform-alerts without paging.

Canary Deployments — Argo Rollouts¶

Opting in¶

To enable canary deployments for a service, set in your helm-values.yaml (or helm-values-local.yaml):

rollout:
  enabled: true
  canary:
    steps:
      - setWeight: 20
      - pause: { duration: 5m }
      - setWeight: 50
      - pause: { duration: 10m }

The golden-path Helm chart converts the Deployment to an Argo Rollout resource when rollout.enabled: true.

Auto-rollback via ClusterAnalysisTemplate¶

A platform-wide ClusterAnalysisTemplate named http-error-rate is deployed at local/argocd/argo-rollouts-values.yaml and aws/argocd/argo-rollouts-values.yaml. It automatically aborts and rolls back the canary if:

5xx error rate exceeds 1% during any analysis interval, or
p99 latency exceeds 500 ms

Scaffolding a canary service¶

Use the Canary Deployment Backstage template (Create → Canary Deployment) to add canary configuration to an existing service. The template writes helm-values-staging.yaml and wires the analysis template.

Accessing Argo Rollouts UI¶

kubectl argo rollouts dashboard -n services-dev

Or via ingress: http://argo-rollouts.idp.local — installed by bootstrap-local.sh (no extra flag), and listed by ./scripts/bootstrap-local.sh --print-urls.

Multi-Environment GitOps Promotion¶

Flow¶

dev (auto)  →  staging (PR)  →  prod (manual approval)

dev: Every merge to main auto-deploys to the services-dev namespace via ArgoCD (idp-services ApplicationSet).
staging: A CI job (promote-to-staging) opens a PR that updates helm-values-staging.yaml with the new image SHA. Merging the PR triggers ArgoCD sync to services-staging (idp-services-staging ApplicationSet).
prod: Once the staging deploy passes its smoke test (smoke-test-staging), the promote-to-production job opens a PR that updates helm-values-prod.yaml. A human must merge it — there is no auto-merge to production. Merging triggers ArgoCD sync to services-prod (idp-services-prod ApplicationSet).

Both idp-services-staging and idp-services-prod use a files generator (not directories) in aws/argocd/app-of-apps.yaml — an Application is only created once the corresponding helm-values-<env>.yaml actually exists for a service, so services that haven't been promoted yet don't produce broken/OutOfSync Applications. The prod ApplicationSet also disables selfHeal so an incident rollback via argocd app rollback isn't immediately reverted by auto-sync.

How `promote-to-staging` works¶

The CI job runs a smoke test after the dev deploy completes. If the smoke test passes, it:

Checks out the platform repo
Updates the image.tag in helm-values-staging.yaml
Opens a PR titled chore: promote <service> <tag> to staging

The PR must be reviewed and merged manually — there is no auto-merge for staging.

Smoke test in CI¶

The smoke-test job runs after the dev deploy and hits GET /healthz on the deployed service. A non-200 response fails the CI run and blocks the staging promotion PR from being opened.

Per-Team Cost Budgets¶

Overview¶

Monthly USD budgets are declared in catalog.teamMetadata (backstage/app-config.yaml) and merged onto each GitHub-Org-synced Group entity's annotations at catalog-sync time (see ADR-0004); PrometheusRules enforce them. Actual costs are queried from OpenCost every 15 minutes by the tech-insights-exporter.

Current budget values¶

Team	Monthly budget (USD)
platform-team	$2,000
ml-team	$1,500
data-team	$800
backend-team	$600
frontend-team	$400
android-team	$300
ios-team	$300
qa-platform-team	$200

Prometheus metrics¶

Metric	Description
`idp_team_budget_usd_monthly{team}`	Configured monthly budget
`idp_team_actual_cost_usd_monthly{team}`	Actual OpenCost spend (updated every 15 min)
`idp_team_budget_utilization_ratio{team}`	`actual / budget` — alerts fire at 0.8 and 1.0

Alerts¶

Alert	Threshold	Severity
`TeamBudgetWarning`	Utilisation > 80%	Warning → Slack
`TeamBudgetExceeded`	Utilisation > 100%	Critical → PagerDuty

See the Cost Budget Exceeded runbook for remediation steps.

Updating a team's budget¶

Edit the team's entry in catalog.teamMetadata (backstage/app-config.yaml):

catalog:
  teamMetadata:
    backend-team:
      costBudgetMonthlyUsd: "1000"

Update the matching entry in kubernetes/finops/team-budgets-configmap.yaml.
Commit and push — the exporter picks up the new value on the next 15-minute cycle.

KAgent Guardrails and Audit Log¶

Structured audit log¶

Every MCP tool call on the idp-mcp-server and contract-mcp-server emits a structured [AUDIT] line to stdout:

[AUDIT] {"ts":"2026-06-09T12:00:00Z","server":"idp-mcp-server","action":"scaffold_service_requested","agent":"idp-assistant","service":"demo-svc","template":"python-service","dry_run":false}

Fields:

Field	Description
`ts`	ISO-8601 timestamp
`server`	MCP server name (`idp-mcp-server`, `contract-mcp-server`)
`action`	Tool-specific action identifier
`agent`	Agent ID (from `X-Agent-ID` header or `User-Agent`)
Tool-specific fields	e.g. `service`, `template`, `dry_run`, `provider`, `version`

Querying audit logs in Loki¶

{app="idp-mcp-server"} |= "[AUDIT]" | json
{app="contract-mcp-server"} |= "[AUDIT]" | json | action="register_contract_requested"

Per-agent metrics in Prometheus/Grafana¶

# Tool call rate per agent
rate(mcp_agent_tool_calls_total{server="idp-mcp-server"}[5m])

# Error rate per tool
rate(mcp_tool_calls_total{outcome="error"}[5m]) /
rate(mcp_tool_calls_total[5m])

The Grafana AI Platform dashboard (http://grafana.idp.local/d/ai-platform) shows these metrics broken down by server, tool, and agent.

dry_run mode¶

Pass dry_run: true when calling scaffold_service to get a preview of what would be created without actually creating anything:

User: "dry run: scaffold a Go service called test-svc"

The agent detects the phrase "dry run" and passes dry_run: true to the tool, which returns a preview JSON without making any Backstage scaffolder calls.

KAgent system-prompt guardrails¶

kubernetes/kagent/idp-agent.yaml includes the following guardrail rules:

Rule	Behaviour
9	Announce to the user before performing any destructive operation (scaffold, deploy)
10	Support `dry_run: true` — use it when the user says "dry run", "preview", or "what would happen"
11	Self-check: if `scaffold_service` has been called more than 3 times in the same session, pause and ask the user to confirm intent

Agent rate alerts¶

Two PrometheusRules in the kagent-guardrails group alert on abnormal agent behaviour:

Alert	Condition	Action
`ScaffoldServiceHighRate`	> 5 scaffold calls in 10 minutes	Warning → Slack
`McpToolErrorRateHigh`	> 50% error rate on any MCP tool	Warning → Slack

See the KAgent Guardrails runbook for investigation steps.

SRE & Reliability¶

SLOs and Error Budgets¶

Overview¶

Defining an SLO¶

How SLOs reach the cluster¶

Multi-window burn-rate alerts¶

Viewing SLO status in Grafana¶

Backstage Metrics¶

PodDisruptionBudgets¶

What ships by default¶

What maxUnavailable: 1 means in practice¶

Overriding¶

Zone spread¶

Blameless Postmortem Process¶

Process¶

Incident Record Automation¶

OPA Cost-Tag Enforcement¶

Log Aggregation — Loki + Promtail¶

Architecture¶

Querying logs in Grafana Explore¶

TraceID linking to Tempo¶

Distributed Tracing — Grafana Tempo¶

Endpoints¶

Instrumenting a service¶

Viewing traces¶

Datadog Infra Observability & APM¶

Coexistence with Prometheus/Grafana¶

Deployment¶

APM traces from the Backstage backend¶

Catalog UI and per-service opt-in¶

Deployment tracking¶

Alert Routing¶

PagerDuty Escalation¶

Configuration¶

What pages on-call¶

What goes to Slack only¶

Canary Deployments — Argo Rollouts¶

Opting in¶

Auto-rollback via ClusterAnalysisTemplate¶

Scaffolding a canary service¶

Accessing Argo Rollouts UI¶

Multi-Environment GitOps Promotion¶

Flow¶

How promote-to-staging works¶

Smoke test in CI¶

Per-Team Cost Budgets¶

Overview¶

Current budget values¶

Prometheus metrics¶

Alerts¶

Updating a team's budget¶

KAgent Guardrails and Audit Log¶

Structured audit log¶

Querying audit logs in Loki¶

Per-agent metrics in Prometheus/Grafana¶

dry_run mode¶

KAgent system-prompt guardrails¶

Agent rate alerts¶

What `maxUnavailable: 1` means in practice¶

How `promote-to-staging` works¶