Skip to content

AWS Deployment Guide

Last Updated: 2026-05-24 Status: Production-Ready

Table of Contents

  1. Pre-Deployment Checklist
  2. Deployment Steps
  3. Post-Deployment Validation
  4. Adding AWS CD to a Scaffolded Service
  5. Known Issues & Fixes
  6. Troubleshooting
  7. Cost Optimization
  8. Production Hardening
  9. Cleanup & Destroy

Pre-Deployment Checklist

Required Tools

  • AWS CLI ≥ 2.15 (aws sts get-caller-identity works)
  • Terraform ≥ 1.5
  • kubectl
  • Helm ≥ 3.x
  • Docker (running, with docker buildx support)
  • GitHub CLI (gh) authenticated
  • jq, python3, openssl available

Required Credentials

Credential Where to get it Where it goes
GITHUB_TOKEN https://github.com/settings/tokens (scopes: repo, read:org, read:user) local/.env
AUTH_GITHUB_CLIENT_ID https://github.com/settings/developers → New OAuth App local/backstage/.env
AUTH_GITHUB_CLIENT_SECRET Same OAuth App local/backstage/.env
ANTHROPIC_API_KEY https://console.anthropic.com/settings/keys Created in Secrets Manager by setup.sh
DD_API_KEY / DD_APP_KEY (optional) https://app.datadoghq.eu/organization-settings/api-keys terraform/terraform.tfvars (datadog_api_key / datadog_app_key)

Auto-generated by bootstrap (no manual action needed): - AUTH_SESSION_SECRET — Terraform generates a 64-char random value - BACKSTAGE_CATALOG_TOKEN — bootstrap.sh generates and injects into Secrets Manager - K8S_SERVICE_ACCOUNT_TOKEN — bootstrap.sh reads from the cluster and injects into Secrets Manager

GitHub OAuth App Setup

Create the OAuth app before running bootstrap so you have the credentials ready:

  1. Go to https://github.com/settings/developers → New OAuth App
  2. Set callback URL to: http://YOUR_BACKSTAGE_ALB_URL/api/auth/github/handler/frame
    (Use a placeholder URL; update it after bootstrap outputs the real ALB hostname)
  3. Copy Client ID and Client Secret into local/backstage/.env

Verify Everything Before Deploying

./scripts/verify-secrets.sh
# Expected: ✅ All critical checks passed!

Deployment Steps

Step 1: Personalize Configuration

./scripts/setup.sh

Replaces moatazeldebsy and other placeholders across all template files, creates .env files, and generates terraform/terraform.tfvars.

Step 2: Bootstrap AWS Infrastructure

./scripts/bootstrap.sh                     # evaluation: terraform.tfvars only, one shared NAT gateway
./scripts/bootstrap.sh --profile medium    # production: per-AZ NAT + RDS Multi-AZ, Karpenter

--profile small|medium|large applies terraform/profiles/<p>.tfvars on top of terraform.tfvars and is remembered for later runs (--profile none clears it). medium and large also install ArgoCD in HA mode (aws/argocd/argocd-ha-values.yaml: Redis HA, two server/repo-server/applicationset replicas with PDBs) and run two Backstage and two Grafana replicas spread across AZs; small and no-profile stay single-replica. On every profile Grafana keeps its state (users, sessions, service-account tokens such as Backstage's GRAFANA_TOKEN) in a dedicated Postgres (terraform/grafana.tf, enable_grafana_db), so it survives pod restarts. Without medium or large, losing the AZ that hosts the single NAT gateway cuts all outbound traffic from the private subnets — see ADR-0009.

What it deploys (in order):

Phase numbers below match the timer_start labels the script prints, so the timing table at the end of a run lines up with this one.

Phase Duration What
1 — Terraform 15–25 min VPC, EKS (6× t3.medium by default; scales to 0 overnight when enable_cost_optimizer = true), RDS (PostgreSQL), ECR, IAM/OIDC, Secrets Manager. EKS and RDS build concurrently, so this is bounded by the slower one, not their sum.
2 — kubectl config <1 min aws eks update-kubeconfig, connectivity check
2.5 — Image builds starts here hello-service + Backstage images build in the background for the rest of the run. Skipped entirely when the source is unchanged and the tag is already in ECR.
3 / 3.6 / 3.7 3–5 min Namespaces, RBAC, External Secrets Operator, ClusterSecretStore, Secrets Manager population
4 — Prometheus + Grafana 4–8 min kube-prometheus-stack, dashboards, PrometheusRules
4a+4b — Pushgateway + OpenCost 1–2 min Installed concurrently
3.8–3.9 / 4.4x / 4.4 3–8 min Gatekeeper + Kyverno, then Rollouts/Loki/Tempo/Datadog, then the exporter group — each a parallel batch
4.5–4.7 — GitOps 3–5 min ArgoCD (with one retry on a slow first ALB), the idp-services ApplicationSet, Crossplane, and the Backstage/ArgoCD tokens. The ApplicationSet auto-discovers services/* and deploys hello-service, idp-mcp-server and qa-mcp-server to services-dev; contract-mcp-server is excluded and deployed only by bootstrap-ai.sh --aws.
5 — Image builds (join) 0–20 min Waits for phase 2.5. Usually already finished, since it has had the whole platform install to run. Then writes the seed image tag into helm-values-aws.yaml.
5.6–5.8 — Backstage 2–8 min ExternalSecret, generated ConfigMaps, deployment, ALB hostname wait, catalog exporter, AlertManager routing
6 — AI/ML platform 8–20 min bootstrap-ai.sh --aws, only with --with-ai/--adp. Argo Workflows and Velero install in parallel with it.

Total: ~40–70 minutes cold. A repeat run against an existing cluster is far shorter — the image builds, helm releases and Terraform providers all skip when nothing has changed. Set IDP_FORCE=1 to override every skip check, and HELM_WAIT_SHORT/HELM_WAIT_MED/HELM_WAIT_LONG/HELM_WAIT_XL to raise the helm timeouts on a slow account — these four now cover every helm --wait in the bootstrap. Defaults are 5m/10m/15m/25m; see Scripts Reference. On a slow link, measure throughput first, then raise them — the defaults turn a slow network into what looks like a hard failure.

Step 3: Update GitHub OAuth Callback URL

After bootstrap completes, the Backstage ALB URL is printed. Update your GitHub OAuth app:

  1. Go to https://github.com/settings/developers → your OAuth app → Edit
  2. Update callback URL: http://<BACKSTAGE_ALB_HOSTNAME>/api/auth/github/handler/frame

Step 4: AI/ML stack — opt-in

AI/ML is a layer, not part of the core, matching how local works. Add it with a flag on the same script:

./scripts/bootstrap.sh --with-ai    # KAgent, MLflow, Langfuse, MCP servers
./scripts/bootstrap.sh --adp        # implies --with-ai, adds the agentic platform

Requires ANTHROPIC_API_KEY in Secrets Manager.

Skipping it is a real cost saving, not just skipped workloads: enable_ai and enable_langfuse gate the Langfuse RDS instance and the MLflow/Langfuse S3 buckets and IRSA roles, so a core-only install never provisions them. The old --skip-ai flag skipped the Helm installs but still built a second RDS instance.

Removing the infrastructure is deliberately explicit: omitting --with-ai on a cluster that already has it keeps it, because a forgotten flag must not drop the Langfuse database. Pass --remove-ai-infra to actually destroy it.

You can also call bootstrap-ai.sh directly — note the --aws flag, without which it targets your local Kind context rather than EKS:

./scripts/bootstrap-ai.sh --aws --adp --region <region> --cluster <cluster-name>

Step 5: Provision team namespaces

After the platform is up, onboard teams via the Backstage Provision Team Namespace template. Before running it, create the per-team ESO IAM role:

cd terraform

# Add each team (repeat for all teams)
terraform apply -var='team_eso_roles=[
  {name="payments", cost_center="CC-1234"},
  {name="platform", cost_center="CC-0001"}
]'

# Get role ARNs to paste into the scaffold form
terraform output team_eso_role_arns

Then in Backstage → Create → Provision Team Namespace → fill in team name, tier, and the IAM role ARN. The scaffold PR creates: - kubernetes/teams/<name>/ with namespace, quota, RBAC, AppProject, ApplicationSet, SecretStore, Grafana folder - backstage/catalog/groups/<name>.yaml — team auto-registers in catalog

Merge the PR. ArgoCD's idp-teams ApplicationSet picks up kubernetes/teams/<name>/ and applies it as the team-<name> Application.

Service path convention: Team service values go under teams/<teamName>/services/<serviceName>/, not services/<teamName>/. See docs/team-management.md for details.

Step 6: GitHub Actions secrets

Service repos scaffolded with Deployment Target: AWS get their AWS_ROLE_ARN secret automatically — a push-only role for that repository's own ECR repository (see Adding AWS CD to a Scaffolded Service). Do not give a service repository the platform's github_actions_role_arn: it only trusts the platform repo, and carries near-admin permissions.

Add these to the platform repo to enable the auto-merge workflow (recommended over a PAT):

Secret Value
APP_ID Numeric GitHub App ID (see docs/github-app-setup.md)
APP_PRIVATE_KEY PEM contents of the App's private key

Add this to the platform repo to enable Datadog deployment markers in build-and-deploy.yml (optional — the pipeline runs fine without it, just skips the marker step):

Secret Value
DD_API_KEY Datadog API key — https://app.datadoghq.eu/organization-settings/api-keys

Rebuilding Backstage

There is no separate Backstage deploy step — bootstrap.sh already did it. Phase 5.6 builds the image, pushes it to ECR under a content-hash tag, applies aws/backstage/deployment.yaml with that tag substituted, waits for the ExternalSecret to sync and for the ALB hostname, then patches the config with the real URLs.

To rebuild and roll out after changing Backstage, re-run the bootstrap — the image fingerprint changes, so it rebuilds and redeploys, and skips everything else that is unchanged:

./scripts/bootstrap.sh --region <region> --cluster-name <name>

Post-Deployment Validation

./scripts/validate-deployment.sh

Runs ~40 automated checks across 10 categories. All should pass.

Component checks

kubectl get pods -n services              # hello-service running
kubectl get pods -n monitoring            # prometheus, grafana, alertmanager, pushgateway
kubectl get pods -n external-secrets      # external-secrets operator
kubectl get clustersecretstore            # aws-secretsmanager → Ready
kubectl get pods -n crossplane-system     # crossplane + 5 provider-aws-* pods
kubectl get providers.pkg.crossplane.io   # all five INSTALLED=True HEALTHY=True
kubectl get ingress -n services           # ALB address

Visit the ALB hostname:

{"service":"hello-service","version":"<sha>","message":"Hello from the IDP!"}

bootstrap.sh installs the full kube-prometheus-stack (Prometheus + Grafana + AlertManager + Pushgateway) at parity with the local Kind setup. Grafana is pre-configured with the CloudWatch datasource using IRSA — no static AWS credentials needed. OPA/Gatekeeper enforces all five golden-path policies (require-health-probes, require-resource-limits, require-labels, deny-latest-tag, require-cost-tags).

Manual Smoke Tests

# Get all ALB URLs
kubectl get ingress -A --no-headers | awk '{print $2, $4}'

# Test Backstage root (should return HTTP 200 with HTML)
curl -I http://<BACKSTAGE_ALB>

# Verify catalog has entities (should return items array)
curl -s http://<BACKSTAGE_ALB>/api/catalog/entities?limit=5 | python3 -c "import sys,json; d=json.load(sys.stdin); print(f'Entities: {len(d)}')"

# Verify DORA metrics in Prometheus
kubectl port-forward -n monitoring svc/prometheus-kube-prometheus-prometheus 9090:9090 &
curl -s "http://localhost:9090/api/v1/query?query=dora_deploy_frequency_per_day" | python3 -c "
import sys,json; d=json.load(sys.stdin); print('DORA series:', len(d.get('data',{}).get('result',[])))"

# Verify QA metrics in Grafana
curl -s "http://localhost:9090/api/v1/query?query=e2e_pass_rate" | python3 -c "
import sys,json; d=json.load(sys.stdin); print('QA series:', len(d.get('data',{}).get('result',[])))"

Expected Service URLs

Backstage has its own NLB. Every other UI is a listener port on one shared ALB (IngressGroup idp-platform), so they share a hostname and differ by port. bootstrap.sh prints them all at the end. To read one later:

source scripts/lib.sh
alb_ingress_url argocd-server argocd          # -> <shared-alb>:8080
Service URL Lookup
Backstage http://<backstage-nlb> kubectl get svc backstage -n backstage -o jsonpath='{.status.loadBalancer.ingress[0].hostname}'
ArgoCD http://<shared-alb>:8080 alb_ingress_url argocd-server argocd
Grafana http://<shared-alb>:3000 (:443 HTTPS with domain_name) alb_ingress_url prometheus-grafana monitoring
hello-service http://<shared-alb>:8081 alb_ingress_url hello-service-dev-service-template services-dev
agent-event-router http://<shared-alb>:8082/webhook/github (only that path is published) AI stack
KAgent UI / IDP Assistant http://<internal-alb>:8083 / :8084 — VPC only AI stack
MLflow / Langfuse / Argo Workflows http://<internal-alb>:5000 / :3001 / :2746 — VPC only AI stack

KAgent, the IDP Assistant A2A endpoint, MLflow, Langfuse and Argo Workflows have no authentication of their own (or, for Argo Workflows, take a bearer token over plain HTTP), so they sit on a second, internal ALB (group.name: idp-internal) rather than the internet-facing idp-platform one. Reach them from inside the VPC (VPN, bastion, SSM port forwarding) or with kubectl port-forward, e.g. kubectl port-forward -n kagent svc/kagent-ui 8083:8080. alb_ingress_url returns their internal hostname the same way.

Adding AWS CD to a Scaffolded Service

Nothing to add. Scaffold the service with Deployment Target: AWS and the template sets it up:

  1. idp:provision-ecr creates the ECR repository <cluster>/<service> and an IAM role, <cluster>-svc-push-<service>, that only the new repository's main branch can assume and that can only push to that ECR repository (Terraform's <cluster>-service-image-push-boundary caps what such roles may ever do).
  2. The role's ARN is written to the new repository as the AWS_ROLE_ARN secret.
  3. The GitOps pull request adds services/<service>/helm-values-aws.yaml to the platform repo, pointing at that ECR repository.

From then on the service's ci.yml pushes every main build to GHCR and to ECR (SHA and latest tags); with a GH_PAT secret it also bumps the tag in the platform repo. ArgoCD deploys it — no Helm or kubectl from the service's CI.

The platform's own github-actions role is deliberately not used here: it trusts only the platform repository and carries near-admin permissions.

Known Issues & Fixes

All issues below are already fixed in the codebase. This section is history, not live advice — it is here so a symptom on an older checkout is searchable.

For the failures found during the 2026-08 AWS runs, grouped by the class of mistake that allowed them and naming the file that now prevents each one, see AWS install: known failure modes. For debugging a run right now, start at TROUBLESHOOTING.md.

Issue 1: Backstage shows 404 at root /

Symptom: Navigating to the Backstage ALB URL shows "Looks like someone dropped the mic!" 404.

Root cause: Backstage config arrays replace rather than merge across config files. The production ConfigMap's extensions list overwrites the base config's page:catalog: path: / entry.

Fix (already applied): backstage/app-config.aws.yaml (rendered into the backstage-config ConfigMap by bootstrap.sh) now explicitly includes page:catalog: path: / alongside page:kubernetes: disabled: true.

Issue 2: Backstage catalog returns 401 / guest login broken

Symptom: All API calls return 401. Guest sign-in doesn't work.

Root cause: dangerouslyDisableDefaultAuthPolicy: true must be under backend.auth, not the top-level auth key.

Fix (already applied): Correct placement in backstage/app-config.aws.yaml: 

backend:
  auth:
    dangerouslyDisableDefaultAuthPolicy: true

Issue 3: Kubernetes tab shows 401 UNAUTHORIZED

Symptom: hello-service Kubernetes tab shows UNAUTHORIZED_ERROR, status code: 401 for all resources.

Root cause: K8S_SERVICE_ACCOUNT_TOKEN in Secrets Manager was REPLACE_ME — never populated with the real cluster token.

Fix (already applied): bootstrap.sh reads the token from backstage-sa-token secret and injects it into Secrets Manager automatically.

Issue 4: ClusterSecretStore fails — external-secrets-sa not found

Symptom: kubectl get clustersecretstore aws-secretsmanager shows InvalidProviderConfig. DORA exporter secret never syncs.

Root cause: ESO Helm chart creates a ServiceAccount named external-secrets, but the IAM trust policy and ClusterSecretStore both reference external-secrets-sa (which doesn't exist by default).

Fix (already applied): bootstrap.sh explicitly creates external-secrets-sa and annotates it with the IRSA role before applying the ClusterSecretStore.

Issue 5: DORA metrics missing from Grafana

Symptom: DORA Metrics dashboard shows "No data".

Root cause: dora-exporter.py was writing to CloudWatch only, but the Grafana dashboard queries Prometheus for dora_deploy_frequency_per_day.

Fix (already applied): Exporter now pushes to Pushgateway using the exact metric names the dashboard expects. CloudWatch publishing is retained as a secondary path.

Issue 6: hello-service ImagePullBackOff on EKS

Symptom: hello-service pods show ImagePullBackOff or no match for platform in manifest.

Root causes: - Image was in private GHCR (403 Forbidden) - Image built on Apple Silicon (arm64) but EKS nodes are amd64

Fix (already applied): helm-values-aws.yaml points to ECR. Build for the correct platform: 

cd services/hello-service
docker buildx build --platform linux/amd64 \
  -t "${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com/idp-mvp/hello-service:latest" \
  --push .

Issue 7: TechDocs 404 for hello-service

Symptom: hello-service Docs tab shows "no index.md in the root of the docs directory".

Root cause: dir:../../../services/hello-service relative traversal from a GitHub-URL-loaded catalog entity is unreliable.

Fix (already applied): backstage/catalog/services/hello-service/catalog-info.yaml uses an explicit GitHub URL: 

backstage.io/techdocs-ref: url:https://github.com/moatazeldebsy/backstage-platform-template/tree/main/services/hello-service

Issue 8: ArgoCD idp-services ApplicationSet missing

Symptom: Services don't appear in ArgoCD. platform-services app shows ComparisonError: app path does not exist.

Root cause: The idp-services ApplicationSet (which auto-discovers services/*/) was never applied.

Fix (already applied): bootstrap.sh Phase 4.6 applies aws/argocd/app-of-apps.yaml.

Issue 9: OpenCost and Pushgateway have no ALB ingress

Symptom: No public URL for OpenCost or Pushgateway after deployment.

This is intentional, not a fault. Both used to publish their own internet-facing ALB (~$16/mo each, no authentication in front of either). They are operator tools — OpenCost's data reaches users through the Backstage FinOps tab and the Grafana FinOps dashboard, and Pushgateway is written to by CI and scraped by Prometheus in-cluster. Reach the raw UIs with:

kubectl port-forward -n opencost   svc/opencost               9090:9090
kubectl port-forward -n monitoring svc/prometheus-pushgateway 9091:9091

The same applies to Prometheus, Alertmanager and the Argo Rollouts dashboard; bootstrap.sh prints every port-forward command in its closing banner.

Issue 10: Tech Radar shows empty / "not allowed to read"

Symptom: Tech Radar page is blank or shows a reading permission error.

Root cause: backend.reading.allow was missing from the ConfigMap, so Backstage blocked reads from raw.githubusercontent.com.

Fix (already applied): backstage/app-config.aws.yaml includes: 

backend:
  reading:
    allow:
      - host: raw.githubusercontent.com
      - host: github.com

Issue 11: Grafana login fails with multiple replicas

Symptom: Logging into Grafana succeeds but immediately redirects to login again.

Root cause: 2 Grafana replicas with ALB (no sticky sessions) — session created on pod A, next request hits pod B.

Fix (already applied): aws/observability/prometheus-stack-values.yaml sets replicas: 1.

Troubleshooting

For a comprehensive fresh-setup troubleshooting guide covering all AWS phases (Terraform, EKS, Backstage, ArgoCD, Crossplane, Observability), see docs/TROUBLESHOOTING.md.

Backstage pod not starting

kubectl logs -n backstage deployment/backstage --tail=50 | grep -E "error|Error|FATAL"
kubectl describe pod -n backstage -l app=backstage | grep -A10 "Events:"

Common causes: - Database not ready: check kubectl get pods -n backstage | grep postgres - Secret not synced: kubectl get externalsecret backstage-secrets -n backstage - Config YAML parse error: check logs for YAMLException

Stale Terraform lock

cd terraform
terraform force-unlock <lock-id>

EKS nodes not ready

kubectl get nodes
kubectl describe nodes | grep -A5 "Conditions:"
kubectl wait --for=condition=Ready node --all --timeout=600s

Re-seed QA metrics after Pushgateway restart

PUSHGATEWAY_URL=http://$(kubectl get ingress prometheus-pushgateway -n monitoring \
  -o jsonpath='{.status.loadBalancer.ingress[0].hostname}') \
  ./scripts/seed-qa-metrics.sh

Trigger DORA metrics collection manually

kubectl create job dora-manual-$(date +%s) --from=cronjob/dora-exporter -n monitoring
kubectl logs -n monitoring -l job-name=dora-manual-* --tail=20

Force Backstage catalog refresh

kubectl rollout restart deployment/backstage -n backstage
# Catalog repopulates within ~60 seconds

Cost Optimization

Free-tier reality

EKS is not free-tier compatible. Fixed monthly costs that cannot be eliminated while the cluster is running:

Resource Monthly floor Free tier?
EKS control plane ~$73 No
NAT Gateway ~$33 + data No
Secrets Manager (5 secrets) ~$2 No

Lowest cost approach: use ./scripts/bootstrap-local.sh --full for daily development. Deploy to AWS only for demos, then clean up immediately.

Default cost-optimized settings

terraform/terraform.tfvars.example, which setup.sh copies on first run, is tuned for the lowest spend that still actually runs the platform:

node_group_min_size      = 0        # scale-to-zero allowed (cost optimizer)
node_group_desired_size  = 6
node_group_max_size      = 8
enable_cost_optimizer    = true     # overnight Lambda scaler (see below)
budget_monthly_limit_usd = "100"    # SNS alert at $80 actual / $100 forecasted

This used to recommend desired_size = 1. A single t3.medium is 2 vCPU / 4 GiB, and rendering every chart and manifest this repo installs gives roughly 2.4 vCPU and 5.7 GiB of declared requests before per-node DaemonSet overhead (~0.3 vCPU and ~900 MiB each, for aws-node, kube-proxy, ebs-csi-node, promtail and the Datadog agent) — so one node could never have run it, and following that advice produced a cluster stuck in Pending.

Six is deliberately conservative rather than measured on a live cluster. Once the platform is up, check the real numbers before going lower — OpenCost and Prometheus are both deployed, and the Grafana FinOps dashboard reports per- namespace cost and efficiency. Note that within the t3 family the $/vCPU and $/GiB are identical, so only the node count changes the bill; moving to t3.large buys memory at exactly proportional cost.

Overnight node scaler

When enable_cost_optimizer = true, two EventBridge-triggered Lambdas run on schedule:

Lambda Schedule (UTC) Action
EKS node scaler 8 PM daily Scale node group → 0
RDS scheduler 8 PM daily Stop RDS instance
Both 7 AM daily Scale back to desired / start RDS

This saves ~60% of EC2 and RDS costs (assuming 9 h active / 24 h). To adjust the schedule:

# terraform/terraform.tfvars
cost_optimizer_scale_down_cron = "cron(0 22 * * ? *)"  # 10 PM UTC
cost_optimizer_scale_up_cron   = "cron(0 6  * * ? *)"  # 6 AM UTC

To disable entirely (always-on cluster): 

enable_cost_optimizer = false

Prometheus storage (already optimized)

aws/observability/prometheus-stack-values.yaml ships with cost-optimized defaults:

prometheusSpec:
  retention: 3d            # time-based cap — reduce scrape history
  retentionSize: 15GB      # size-based cap — this is the load-bearing one
  storageSpec:
    volumeClaimTemplate:
      spec:
        storageClassName: gp3   # 20% cheaper than gp2
        resources:
          requests:
            storage: 20Gi       # headroom above retentionSize

Do not drop storage below retentionSize with room to spare, and do not remove retentionSize. retention alone is a time cap, so a busy cluster can fill the volume long before the window expires — a previous 5Gi volume filled in about four days and silently wedged ingestion. retentionSize is what actually bounds the volume; storage: 20Gi exists to give it headroom.

For longer history, raise both together (e.g. retention: 15d, retentionSize: 45GB, storage: 60Gi) and expect the EBS cost to rise with it.

Skip the AI/ML stack

KAgent + MLflow + MCP servers add ~6 platform ALBs plus one per MCP server per environment (~$36–72/month for the platform pieces alone, more once the MCP servers are promoted to staging and prod) and ~6 Gi EBS. Skip them unless needed:

# Core platform only (no AI/ML)
./scripts/bootstrap.sh

# Add AI/ML later when needed
./scripts/bootstrap-ai.sh --aws
# Remove it
./scripts/bootstrap-ai.sh --aws --destroy

Estimated monthly cost by configuration

Rough us-east-1 on-demand estimates at ~730 h/month. Treat them as order-of- magnitude only — the authoritative numbers are the AWS budget alert below and the OpenCost/Grafana FinOps dashboard, both of which this platform deploys.

The fixed floor is ~$135/mo before a single workload runs: EKS control plane (~$73), one NAT gateway (~$33), RDS db.t3.micro (~$17), plus EBS/S3/ECR/ Secrets (~$15). Nodes and ALBs are what actually move.

Config EC2 nodes Cost optimizer AI/ML ALBs Est. monthly
Default 6× t3.medium On (8 PM–7 AM UTC) Off 3 ~$280–310
Always-on, no AI 6× t3.medium Off Off 3 ~$370–400
Always-on + AI 6× t3.medium Off On 6 ~$430–490
Production per profiles/medium.tfvars Off Optional 6 ~$700–900

The ALB column understates a full AI deployment. Measured on the live cluster 2026-08-17, 27 ALB Ingresses existed with --adp and all three environments populated (dev/staging/prod each publish their own), not 6 — one per Ingress, since group.name is unused. Removing the six dead internal ones below brings that to 21. Treat the table as the core-platform baseline.

Scaffolded services no longer add to this by default: since #397 their helm-values-aws.yaml ships ingress.enabled: false, so a new service publishes no ALB unless you opt in. The 27 measured above predate that change and reflect environments whose services all had ingress enabled.

Where the money goes, and what has already been done about it:

  • Nodes dominate — 6× t3.medium is ~$182/mo on-demand. enable_cost_optimizer takes ~45% off that by scaling to zero overnight. Within the t3 family the $/vCPU and $/GiB are identical, so only the node count matters, not the size.
  • ALBs used to be the silent second-biggest line: every ALB Ingress provisions its own load balancer at ~$16/mo, and there were eleven. Prometheus, Alertmanager, Pushgateway, OpenCost and the Argo Rollouts dashboard no longer publish one — they are operator tools, reachable with kubectl port-forward (the bootstrap banner prints the exact commands). That is ~$82/mo, and it also took five unauthenticated endpoints off the public internet.
  • Six internal ALBs were pure waste and have been removed (2026-08-18). approval-service and the argocd/cost/github/incident/security MCP servers each published an internal ALB with a <service>.internal hostname. No private hosted zone exists, so those names resolved nowhere, and every real caller uses in-cluster Service DNS anyway — app-config.aws.yaml proxies straight to *.services-dev.svc.cluster.local. They routed nothing and billed ~$96/mo. Their ingress.enabled is now false, which is also more private than an internal ALB.
  • Consolidating the eight remaining internet-facing ALBs onto one would save ~$112/mo more. See Consolidating ALBs — it needs real DNS, so it is a project rather than a flag.

Measured: a full six-node cluster with AI/ML

Measured against a running idp-mvp cluster in us-east-1 on 2026-08-14, at on-demand list prices, for continuous 24/7 running — that is, before the overnight scale-down enable_cost_optimizer performs by default. Local is free — this is only the EKS path. These are AWS infrastructure charges only; see what the measurement excludes below before treating the total as your bill.

This is a measurement, not the repo default. The table reflects a running idp-mvp cluster with the node group scaled up to six t3.large. terraform/variables.tf ships node_group_desired_size = 1 (min 0, max 2), so a bootstrap using the defaults costs substantially less — and will not fit the full stack. Scale the node group deliberately rather than inferring it from this table.

Component Qty ~$/month
EKS control plane 1 73
Worker nodes (t3.large) 6 364
NAT gateway 1 33 + data
Application Load Balancers — core 4 66
RDS for Backstage (db.t3.micro) 1 12
S3 / ECR / Secrets Manager / CloudWatch — ~15
Core platform subtotal ~565
Application Load Balancers — AI/ML 11 181
RDS for Langfuse (db.t4g.micro) 1 11
S3 for MLflow + Langfuse artifacts 2 ~2
AI/ML layer subtotal (--with-ai) ~195
Total with AI/ML ~760

The load balancers are the surprise. Every ALB is ~$16/month before traffic, and the AI/ML layer creates eleven of the fifteen — one per MCP server, plus KAgent, the IDP assistant, MLflow and Langfuse. That is more than the RDS instances and S3 combined, and it is why the AI layer is opt-in (./scripts/bootstrap.sh --with-ai) rather than default.

Ways to spend less, roughly in order of effect:

  • Already on by default: spot nodes for every non-prod install (node_capacity_type, ~60–70% off the node line), and one shared ALB for all platform UIs instead of one each (see Consolidating ALBs). medium/large stay on-demand. Keep cluster_version in EKS standard support too: an extended-support version bills the control plane at 6×.
  • Skip the AI/ML layer. Saves ~$195/month. enable_ai and enable_langfuse gate the infrastructure too, so nothing is provisioned for it.
  • Leave enable_cost_optimizer = true (the default). Scales nodes to zero and stops RDS overnight — roughly halves the node and RDS lines if you only work office hours. Because it is on by default, a stock deployment on an office-hours schedule already lands well under the table above, which measures the optimizer-disabled case.
  • Drop the node count. Six t3.large is sized for the full stack including AI; the core platform alone fits in fewer. Note the constraint documented on node_instance_types: nodes are sized by pod IP capacity, not CPU/RAM.
  • Tear down when idle. ./scripts/cleanup.sh, not a bare terraform destroy. Several things the platform runs are created by in-cluster controllers rather than Terraform: ALBs, the EBS disks behind PersistentVolumes, Crossplane resources. They survive terraform destroy and keep billing. On one account the PVC disks alone kept ~$13/month running after the cluster was gone. cleanup.sh deletes them, and its final verification reports anything left over. The full list is in what Terraform cannot destroy.

What the measurement excludes

The total is the platform's own AWS footprint. It is not an all-in run rate:

Not counted Why it can matter
Data transfer and NAT data processing Charged per GB on top of the NAT hourly rate. Image pulls and cross-AZ traffic dominate it, so it scales with your workload, not with the platform
LLM API spend KAgent runs Claude and GPT-4o, and AI Search needs a VOYAGE_API_KEY. Those are Anthropic / OpenAI / Voyage bills, not AWS — and on an agent-heavy platform they can exceed the infrastructure. This is precisely what the Langfuse page exists to show you
Datadog Third-party SaaS priced per host and per ingested GB, alongside the Prometheus/Grafana stack that is included
Your own services Everything above is the platform. Whatever your teams scaffold onto it is additional
Savings Plans / Reserved Instances List prices only. Committed-use discounts take a meaningful cut off the node line

Verify against the AWS pricing calculator for your region before committing — this table is a measurement of one cluster, not a quote.

Consolidating ALBs

Done without a domain: one ALB, one listener port per service. Every platform Ingress carries alb.ingress.kubernetes.io/group.name: idp-platform and its own listen-ports (ArgoCD :8080, Grafana :3000, hello-service :8081, agent-event-router :8082; the unauthenticated AI-stack UIs :2746–:8084 use the same pattern on the internal idp-internal group — see the URL table above). Rules live per listener, so the host: "" + path: / entries no longer collide. That replaced ~9 ALBs, each ~$0.0375/h with its three public IPv4 addresses, with one. validate-deployment.sh flags any internet-facing ALB Ingress that is missing the group annotation. To add one, pick an unused port, set both annotations, and read the URL with alb_ingress_url.

Scaffolded services' own helm-values-aws.yaml are not part of the group: a new service gets its own ALB until it is given a port here, or a hostname as below.

The host-based setup below is the upgrade path once a domain exists. It gives port-free URLs and real TLS for every service. Without a domain, it would need the hostnames to differ, which they cannot:

One ALB is not achievable while keeping internal services private. An IngressGroup maps to exactly one ALB with one scheme, so internal and internet-facing Ingresses cannot share a group. The realistic floor is one shared internet-facing ALB; anything that must stay private should publish no Ingress at all and be reached in-cluster, which is what the six removed above now do.

Steps, in order:

  1. Set domain_name in terraform/terraform.tfvars (a Route 53 zone must already exist for it). terraform/acm.tf is already gated on this and will provision a wildcard *.${domain_name} certificate with DNS validation — no new Terraform is needed for TLS.
  2. Give every internet-facing service a hostname in its helm-values-aws.yaml, e.g. host: hello-service.idp.example.com.
  3. Add a shared group to those same annotations: 
    alb.ingress.kubernetes.io/group.name: idp-public
alb.ingress.kubernetes.io/group.order: "10"   # lower = evaluated first
alb.ingress.kubernetes.io/listen-ports: '[{"HTTP":80},{"HTTPS":443}]'
alb.ingress.kubernetes.io/certificate-arn: <wildcard cert ARN from acm.tf>
    Every Ingress carrying the same group.name merges onto one ALB.
  4. Point DNS at it — see below.

DNS: wildcard now, external-dns at scale

For this platform, start with a single wildcard ALIAS: one *.idp.example.com record to the shared ALB, matching the wildcard certificate acm.tf already issues. No controller, no extra IAM, and every future scaffolded service resolves the moment its Ingress joins the group. For a template repo that others adopt, that is the right default because it adds no prerequisites.

At organisation scale, the standard is external-dns over a delegated subdomain. The reason is lifecycle, not convenience: records are created and deleted with the Ingress, so a scaffolded service needs no ticket and no Terraform PR. If you adopt it:

  • Delegate, never grant apex access. NS-delegate dev.idp.example.com to a zone the platform owns. A controller with write access to the company apex zone is an unacceptable blast radius.
  • Scope the IRSA role to that hosted zone ARN only.
  • Set --txt-owner-id=<cluster-name> so several clusters can share a zone without overwriting each other's records. This is what makes it multi-cluster safe, and it is the setting most often missed.
  • Choose --policy deliberately. sync deletes records when the Ingress goes; upsert-only is safer on day one but leaks records forever, which becomes its own cleanup project.
  • Pair it with the wildcard cert or cert-manager. DNS automation without matching certificate automation just moves the ticket.

Further cost optimisations (proposed, not yet implemented)

Measured against the real August 2026 bill, for a cluster that ran six days (2026-08-12 to 08-18) at $45–51/day:

  79.04  EKS control plane        (fixed, $0.10/hr per cluster)
  67.99  EC2 - Compute            (the nodes)
  56.26  Elastic Load Balancing   (one ALB per Ingress)
  29.02  VPC                      (NAT gateway)
  27.54  EC2 - Other              (EBS)
   5.28  RDS

Already implemented — single NAT gateway, Karpenter with a spot-first NodePool and spot-interruption handling, managed node group scaled to desired_size = 1 with Karpenter taking the burst, overnight scale-to-zero, 7-day EKS log retention, Prometheus storage limits, and eleven ALBs removed (five operator tools, then six dead internal ones).

Ranked by what is actually left:

# Proposal Est. saving Effort / risk
1 Destroy between sessions. Nothing else comes close — the platform costs ~$48/day running and ~$1.60/month destroyed. The state bucket and lock table are preserved precisely so a rebuild is cheap. ~$1,400/mo vs always-on None; already the practice
2 Consolidate the eight internet-facing ALBs onto one host-routed ALB ~$112/mo Medium — needs domain_name, DNS, ACM
3 Drop the NAT gateway in dev. ~$32/mo plus data processing, purely so private nodes can reach the internet. A dev cluster can run nodes in public subnets, or keep NAT and add S3/ECR/STS VPC endpoints so image pulls stop crossing it. $29–32/mo Medium; endpoints have their own hourly cost, so measure first
4 Force gp3 EBS. No volume_type is set anywhere in terraform/, so volumes land on the driver default. gp3 is ~20% cheaper than gp2 and decouples IOPS from size. ~$5/mo at current usage Low — a StorageClass change
5 Set retention on non-EKS log groups. Only eks.tf sets retention_in_days; the Lambda log group had none, meaning never expire. Small now, grows forever Low
6 Right-size the memory-optimized node group. It defaults to desired_size = 0 and only scales for AI workloads — confirm it returns to zero after bootstrap-ai.sh runs, rather than idling. Up to ~$60/mo if it idles Low — verification, not code

Not ours, but the largest residual line in this account once the platform is destroyed: a 50 GB RDS snapshot (infra-landscape-dev-postgres-final-snapshot, ~$4.75/mo) and a disabled KMS key (alias/mokey, created 2021, ~$1/mo) — both left over from an unrelated project. Together they cost roughly 3.5× everything this platform leaves behind.

Budget alert

A budget is provisioned in Terraform at $100/month with SNS alerts at: - 80% actual (~$80 spent) — early warning - 100% forecasted — projected overage

Set budget_alert_email in terraform/terraform.tfvars to receive email alerts directly. The SNS → Slack Lambda bridge fires if idp-mvp/slack-webhook is populated in Secrets Manager.

Production Hardening

Scale EKS for production load

# terraform/terraform.tfvars
node_instance_types     = ["t3.large"]
node_group_min_size     = 2
node_group_desired_size = 3
node_group_max_size     = 8
enable_cost_optimizer   = false   # keep nodes up 24/7 for production

Enable RDS deletion protection and backups

# terraform/terraform.tfvars
environment             = "prod"   # enables deletion_protection=true, skip_final_snapshot=false
# Note: backup_retention_period is currently set to 1 (cost-optimized default).
# For production, increase it in terraform/rds.tf:
#   backup_retention_period = 7

Enable ALB sticky sessions (if scaling Grafana)

Add to Grafana ALB ingress annotations: 

alb.ingress.kubernetes.io/target-group-attributes: stickiness.enabled=true,stickiness.lb_cookie.duration_seconds=86400

Rotate GitHub token

aws secretsmanager get-secret-value --secret-id idp-mvp/backstage --region us-east-1 \
  --query SecretString --output text | python3 -c "
import sys, json
d = json.load(sys.stdin)
d['GITHUB_TOKEN'] = 'ghp_NEW_TOKEN_HERE'
print(json.dumps(d))
" | aws secretsmanager put-secret-value \
  --secret-id idp-mvp/backstage --region us-east-1 --secret-string file:///dev/stdin

kubectl annotate externalsecret backstage-secrets -n backstage force-sync=$(date +%s) --overwrite
kubectl rollout restart deployment/backstage -n backstage

Set PagerDuty / Jira credentials (optional)

Both are optional — left unset, the On-Call and Issues tabs render their empty state rather than erroring, exactly as they do locally. Supply them either through Terraform:

# terraform.tfvars
pagerduty_token = "your-read-only-rest-api-key"
jira_token      = "base64-of-email:api_token"
jira_url        = "https://your-company.atlassian.net"

...or by patching the existing secret in place, which avoids a Terraform apply:

aws secretsmanager get-secret-value --secret-id idp-mvp/backstage --region us-east-1 \
  --query SecretString --output text | python3 -c "
import sys, json
d = json.load(sys.stdin)
d['PAGERDUTY_TOKEN'] = 'YOUR_PAGERDUTY_KEY'
d['JIRA_TOKEN']      = 'YOUR_BASE64_EMAIL_COLON_TOKEN'
d['JIRA_URL']        = 'https://your-company.atlassian.net'
print(json.dumps(d))
" | aws secretsmanager put-secret-value \
  --secret-id idp-mvp/backstage --region us-east-1 --secret-string file:///dev/stdin

kubectl annotate externalsecret backstage-secrets -n backstage force-sync=$(date +%s) --overwrite
kubectl rollout restart deployment/backstage -n backstage

JIRA_TOKEN is base64(email:api_token), not the raw API token. Generate the token at https://id.atlassian.com/manage-profile/security/api-tokens, then printf 'you@example.com:TOKEN' | base64.

Leaving JIRA_URL empty keeps app-config.aws.yaml on its RFC 2606 https://jira.invalid default, which never resolves — deliberate, so an unconfigured proxy cannot send requests to a domain nobody here controls.

Pod Disruption Budgets

kubectl apply -f - <<EOF
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: backstage-pdb
  namespace: backstage
spec:
  minAvailable: 1
  selector:
    matchLabels:
      app: backstage
EOF

Cleanup & Destroy

# Safe teardown — runs all phases in the correct order
./scripts/cleanup.sh --cluster-name idp-mvp --force

cleanup.sh runs nine ordered phases (0–8). The order is the whole point — see why terraform destroy alone is not enough.

Phase What it does
0 — Stop the writers Deletes ArgoCD ApplicationSets, then workload Applications with cascade, so the load balancer controller tears down its own ALBs instead of leaking them. Also scales the Loki write path to zero — Loki is Helm-installed, not ArgoCD-managed, and will otherwise keep writing into the S3 bucket Phase 5 is trying to empty.
1 — ALBs Deletes any orphaned Kubernetes-managed ALBs (k8s-*) and stale k8s-* security groups. Both block VPC deletion, and the security groups only release asynchronously, so this retries.
2 — RDS protection Disables deletion protection on the Backstage RDS instance
3 — Scaffolded services Auto-discovers user-scaffolded services in services/ (excludes built-ins). Deletes their ArgoCD Applications, uninstalls Helm releases, removes services/<name>/ and commits the deletion. Runs while EKS is still up so ArgoCD can cascade cleanly.
4 — Crossplane resources Finds everything tagged idp:provisioner=crossplane via the Resource Groups Tagging API and deletes it: S3 buckets (all versions + delete markers), RDS instances, DynamoDB tables, SQS queues. MSK topics go with the cluster in Phase 6.
5 — S3 + ECR empty Empties Terraform-managed S3 buckets and force-deletes <cluster>-prefixed ECR repos. AWS blocks bucket deletion when non-empty and force_destroy = false, which is the case for every bucket here.
6 — Terraform destroy Destroys all Terraform-managed resources: EKS, VPC, IAM/IRSA, RDS, ECR, KMS, Secrets Manager. Retried, because BucketNotEmpty and security-group DependencyViolation are both expected and self-healing.
7 — CloudWatch Deletes log groups EKS and Lambda create at runtime (/aws/eks/<cluster>, /aws/containerinsights/<cluster>, /aws/lambda/<cluster>-*) — these survive the cluster and keep billing
8 — Verify Checks EKS, RDS, ALBs, Crossplane-tagged resources and log groups are all zero

Preserved deliberately: the Terraform state bucket (<cluster>-terraform-state-*), its DynamoDB lock table (<cluster>-terraform-locks), and any ECR repo whose name does not contain the cluster name. Those bare-named repos were never Terraform-managed, so a teardown leaves them — see the ownership table below.

What Terraform owns, and what it cannot destroy

No, terraform destroy on its own will not clean up this platform, and the gap is not an oversight — it follows from how the platform is built. Terraform owns the account and cluster foundation. Controllers running inside the cluster create AWS resources of their own in response to Kubernetes objects, and Terraform has never heard of those.

Resource Owner Why
EKS, VPC, subnets, NAT, IAM/IRSA, OIDC provider, RDS, KMS, Secrets Manager, budgets, ACM Terraform (terraform/, ~270 resources) Account/cluster foundation, applied once by bootstrap.sh
ECR repos named <cluster>/<service> Terraform (force_delete = true) Declared in ecr.tf; images are removed with the repo
ALBs, target groups, k8s-* security groups AWS Load Balancer Controller Created at runtime from Ingress objects. They exist in no state file, and they block VPC deletion. Delete the Ingress and the controller cleans up after itself; delete the cluster first and they orphan at ~$16/mo each, invisible to Terraform. This is why Phase 0 runs before Phase 6.
S3/RDS/DynamoDB/SQS/MSK from Claims Crossplane Provisioned from Claims committed to Git. Compositions set deletionPolicy: Orphan on purpose — a deleted Claim must not silently destroy a team's data. They are found by the idp:provisioner=crossplane tag instead.
ECR repos created by bootstrap-ai.sh The script, imperatively AI/MCP service repos are created on demand, not declared in Terraform
Contents of S3 buckets and ECR repos Nobody — runtime data Terraform can delete a bucket but AWS refuses while it holds objects and force_destroy = false. Contents must be emptied first, which is Phase 5.
EBS volumes behind PersistentVolumes (Prometheus, Alertmanager, Loki, the MLflow SQLite PVC, …; KAgent's database is RDS when the AI layer is on) EBS CSI driver Created on demand for each PVC and tagged kubernetes.io/cluster/<cluster>=owned. Deleting the cluster detaches them but never deletes them; they keep billing as available volumes (18 volumes / 145 GB had piled up across deploy cycles by 2026-09). Phase 7b deletes them by that tag after terraform destroy.
Karpenter nodes, launch templates, instance profiles (only with enable_karpenter = true) Karpenter Launched by the controller from NodePools, not by Terraform. If the controller is uninstalled while nodes are still terminating they orphan at full EC2 price, and their ENIs block the VPC delete. Phase 0b drains the NodePools while the controller is alive; Phase 5b terminates stragglers before terraform destroy; Phase 7c deletes the launch templates (karpenter.k8s.aws/cluster) and <cluster>_* instance profiles afterwards.
Secret <cluster>/langfuse/project-keys bootstrap-ai.sh, imperatively Created with the CLI when Langfuse is enabled on AWS; deleted in Phase 5b with the same zero-day recovery window Terraform uses for its own secrets.
/aws/eks/*, /aws/lambda/* log groups EKS / Lambda, at runtime Created by the services themselves and outlive the cluster
Kubeconfig contexts Local machine Not an AWS resource; prune with kubectl config delete-context

Anything else still tagged Project=<cluster> or kubernetes.io/cluster/<cluster> after all that is reported by a tag sweep in Phase 8, which then refuses to call the cleanup complete. That is how an orphaned Terraform launch template from an interrupted May destroy would have been caught at the time instead of months later. KMS keys and secrets that are only scheduled for deletion are expected and skipped.

Two more reasons ordering matters, both learned the hard way:

  • KMS keys and Secrets Manager secrets are never deleted immediately. Terraform "destroys" them, but AWS only schedules deletion — 7–30 days for a KMS key, 7–30 for a secret. They keep billing (~$1/mo and ~$0.40/mo) until the window closes. This is AWS behaviour, not a bug in the teardown.
  • Once EKS is gone, Terraform cannot even plan. The kubernetes, helm and kubectl providers in main.tf take their host from module.eks.cluster_endpoint. With the cluster destroyed that value is empty and the provider fails to configure, so a destroy interrupted after the cluster goes but before the VPC does cannot simply be re-run. cleanup.sh detects this and falls back to per-resource -target destroys.

If a teardown stalls, check the object count of the bucket being emptied rather than the log — _empty_versioned_bucket only logs when it finishes, so a slow phase and a wedged one look identical from the output.

Note on Crossplane resources: Because all Compositions use deletionPolicy: Orphan, Crossplane resources survive Claim deletion. Phase 3 of cleanup.sh handles them automatically by querying the idp:provisioner=crossplane tag. RDS deletions are asynchronous — the script initiates them and notes it in the summary.

To tear down only the AI/ML components on AWS without touching the core platform:

./scripts/bootstrap-ai.sh --aws --destroy

Important — local vs AWS separation: bootstrap-ai.sh --destroy scopes deletions to the active environment. Passing --aws removes aws/ ingresses and the MLflow AWS overlay; omitting it removes only local/ ingresses. Never run bootstrap-ai.sh --destroy (without --aws) while KUBECONFIG points at EKS — it will target the wrong cluster.

To tear down only the local Kind/Rancher cluster (no AWS resources are affected):

./scripts/bootstrap-local.sh --destroy

This runs the same scaffolded-service cleanup logic (ArgoCD Applications deleted, Helm releases uninstalled, services/<name>/ directories removed and committed) before the Kind cluster or Rancher Desktop namespaces are deleted.