Skip to content

AWS Deployment Guide

Last Updated: 2026-05-24 Status: Production-Ready

Table of Contents

  1. Pre-Deployment Checklist
  2. Deployment Steps
  3. Post-Deployment Validation
  4. Known Issues & Fixes
  5. Troubleshooting
  6. Cost Optimization
  7. Production Hardening
  8. Cleanup & Destroy

Pre-Deployment Checklist

Required Tools

  • AWS CLI ≥ 2.15 (aws sts get-caller-identity works)
  • Terraform ≥ 1.5
  • kubectl
  • Helm ≥ 3.x
  • Docker (running, with docker buildx support)
  • GitHub CLI (gh) authenticated
  • jq, python3, openssl available

Required Credentials

Credential Where to get it Where it goes
GITHUB_TOKEN https://github.com/settings/tokens (scopes: repo, read:org, read:user) local/.env
AUTH_GITHUB_CLIENT_ID https://github.com/settings/developers → New OAuth App local/backstage/.env
AUTH_GITHUB_CLIENT_SECRET Same OAuth App local/backstage/.env
ANTHROPIC_API_KEY https://console.anthropic.com/settings/keys Created in Secrets Manager by setup.sh
DD_API_KEY / DD_APP_KEY (optional) https://app.datadoghq.eu/organization-settings/api-keys terraform/terraform.tfvars (datadog_api_key / datadog_app_key)

Auto-generated by bootstrap (no manual action needed): - AUTH_SESSION_SECRET — Terraform generates a 64-char random value - BACKSTAGE_CATALOG_TOKEN — bootstrap.sh generates and injects into Secrets Manager - K8S_SERVICE_ACCOUNT_TOKEN — bootstrap.sh reads from the cluster and injects into Secrets Manager

GitHub OAuth App Setup

Create the OAuth app before running bootstrap so you have the credentials ready:

  1. Go to https://github.com/settings/developers → New OAuth App
  2. Set callback URL to: http://YOUR_BACKSTAGE_ALB_URL/api/auth/github/handler/frame
    (Use a placeholder URL; update it after bootstrap outputs the real ALB hostname)
  3. Copy Client ID and Client Secret into local/backstage/.env

Verify Everything Before Deploying

1
2
./scripts/verify-secrets.sh
# Expected: ✅ All critical checks passed!

Deployment Steps

Step 1: Personalize Configuration

1
./scripts/setup.sh

Replaces moatazeldebsy and other placeholders across all template files, creates .env files, and generates terraform/terraform.tfvars.

Step 2: Bootstrap AWS Infrastructure

1
./scripts/bootstrap.sh

What it deploys (in order):

Phase numbers below match the timer_start labels the script prints, so the timing table at the end of a run lines up with this one.

Phase Duration What
1 — Terraform 15–25 min VPC, EKS (6× t3.medium by default; scales to 0 overnight when enable_cost_optimizer = true), RDS (PostgreSQL), ECR, IAM/OIDC, Secrets Manager. EKS and RDS build concurrently, so this is bounded by the slower one, not their sum.
2 — kubectl config <1 min aws eks update-kubeconfig, connectivity check
2.5 — Image builds starts here hello-service + Backstage images build in the background for the rest of the run. Skipped entirely when the source is unchanged and the tag is already in ECR.
3 / 3.6 / 3.7 3–5 min Namespaces, RBAC, External Secrets Operator, ClusterSecretStore, Secrets Manager population
4 — Prometheus + Grafana 4–8 min kube-prometheus-stack, dashboards, PrometheusRules
4a+4b — Pushgateway + OpenCost 1–2 min Installed concurrently
3.8–3.9 / 4.4x / 4.4 3–8 min Gatekeeper + Kyverno, then Rollouts/Loki/Tempo/Datadog, then the exporter group — each a parallel batch
4.5–4.7 — GitOps 3–5 min ArgoCD (with one retry on a slow first ALB), the idp-services ApplicationSet, Crossplane, and the Backstage/ArgoCD tokens. The ApplicationSet auto-discovers services/* and deploys hello-service, idp-mcp-server and qa-mcp-server to services-dev; contract-mcp-server is excluded and deployed only by bootstrap-ai.sh --aws.
5 — Image builds (join) 0–20 min Waits for phase 2.5. Usually already finished, since it has had the whole platform install to run. Then writes the seed image tag into helm-values-aws.yaml.
5.6–5.8 — Backstage 2–8 min ExternalSecret, generated ConfigMaps, deployment, ALB hostname wait, catalog exporter, AlertManager routing
6 — AI/ML platform 8–20 min bootstrap-ai.sh --aws, only with --with-ai/--adp. Argo Workflows and Velero install in parallel with it.

Total: ~40–70 minutes cold. A repeat run against an existing cluster is far shorter — the image builds, helm releases and Terraform providers all skip when nothing has changed. Set IDP_FORCE=1 to override every skip check, and HELM_WAIT_SHORT/HELM_WAIT_MED/HELM_WAIT_LONG/HELM_WAIT_XL to raise the helm timeouts on a slow account — these four now cover every helm --wait in the bootstrap. Defaults are 5m/10m/15m/25m; see Scripts Reference. On a slow link, measure throughput first, then raise them — the defaults turn a slow network into what looks like a hard failure.

Step 3: Update GitHub OAuth Callback URL

After bootstrap completes, the Backstage ALB URL is printed. Update your GitHub OAuth app:

  1. Go to https://github.com/settings/developers → your OAuth app → Edit
  2. Update callback URL: http://<BACKSTAGE_ALB_HOSTNAME>/api/auth/github/handler/frame

Step 4: AI/ML stack — opt-in

AI/ML is a layer, not part of the core, matching how local works. Add it with a flag on the same script:

1
2
./scripts/bootstrap.sh --with-ai    # KAgent, MLflow, Langfuse, MCP servers
./scripts/bootstrap.sh --adp        # implies --with-ai, adds the agentic platform

Requires ANTHROPIC_API_KEY in Secrets Manager.

Skipping it is a real cost saving, not just skipped workloads: enable_ai and enable_langfuse gate the Langfuse RDS instance and the MLflow/Langfuse S3 buckets and IRSA roles, so a core-only install never provisions them. The old --skip-ai flag skipped the Helm installs but still built a second RDS instance.

Removing the infrastructure is deliberately explicit: omitting --with-ai on a cluster that already has it keeps it, because a forgotten flag must not drop the Langfuse database. Pass --remove-ai-infra to actually destroy it.

You can also call bootstrap-ai.sh directly — note the --aws flag, without which it targets your local Kind context rather than EKS:

1
./scripts/bootstrap-ai.sh --aws --adp --region <region> --cluster <cluster-name>

Step 5: Provision team namespaces

After the platform is up, onboard teams via the Backstage Provision Team Namespace template. Before running it, create the per-team ESO IAM role:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
cd terraform

# Add each team (repeat for all teams)
terraform apply -var='team_eso_roles=[
  {name="payments", cost_center="CC-1234"},
  {name="platform", cost_center="CC-0001"}
]'

# Get role ARNs to paste into the scaffold form
terraform output team_eso_role_arns

Then in Backstage → Create → Provision Team Namespace → fill in team name, tier, and the IAM role ARN. The scaffold PR creates: - kubernetes/teams/<name>/ with namespace, quota, RBAC, AppProject, ApplicationSet, SecretStore, Grafana folder - backstage/catalog/groups/<name>.yaml — team auto-registers in catalog

Merge the PR — CI (scaffold.yml) applies the manifests.

Service path convention: Team service values go under teams/<teamName>/services/<serviceName>/, not services/<teamName>/. See docs/team-management.md for details.

Post-Deployment Validation

1
./scripts/validate-deployment.sh

Runs ~40 automated checks across 10 categories. All should pass.

Manual Smoke Tests

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
# Get all ALB URLs
kubectl get ingress -A --no-headers | awk '{print $2, $4}'

# Test Backstage root (should return HTTP 200 with HTML)
curl -I http://<BACKSTAGE_ALB>

# Verify catalog has entities (should return items array)
curl -s http://<BACKSTAGE_ALB>/api/catalog/entities?limit=5 | python3 -c "import sys,json; d=json.load(sys.stdin); print(f'Entities: {len(d)}')"

# Verify DORA metrics in Prometheus
kubectl port-forward -n monitoring svc/prometheus-kube-prometheus-prometheus 9090:9090 &
curl -s "http://localhost:9090/api/v1/query?query=dora_deploy_frequency_per_day" | python3 -c "
import sys,json; d=json.load(sys.stdin); print('DORA series:', len(d.get('data',{}).get('result',[])))"

# Verify QA metrics in Grafana
curl -s "http://localhost:9090/api/v1/query?query=e2e_pass_rate" | python3 -c "
import sys,json; d=json.load(sys.stdin); print('QA series:', len(d.get('data',{}).get('result',[])))"

Expected Service URLs

Service How to get URL
Backstage kubectl get ingress backstage -n backstage -o jsonpath='{.status.loadBalancer.ingress[0].hostname}'
Grafana kubectl get ingress -n monitoring -l app.kubernetes.io/name=grafana -o jsonpath='{.items[0].status.loadBalancer.ingress[0].hostname}'
ArgoCD kubectl get ingress argocd-server -n argocd -o jsonpath='{.status.loadBalancer.ingress[0].hostname}'
hello-service kubectl get ingress -n services-dev -l app.kubernetes.io/instance=hello-service-dev -o jsonpath='{.items[0].status.loadBalancer.ingress[0].hostname}'

Known Issues & Fixes

All issues below are already fixed in the codebase. This section is history, not live advice — it is here so a symptom on an older checkout is searchable.

For the failures found during the 2026-08 AWS runs, grouped by the class of mistake that allowed them and naming the file that now prevents each one, see AWS install: known failure modes. For debugging a run right now, start at TROUBLESHOOTING.md.

Issue 1: Backstage shows 404 at root /

Symptom: Navigating to the Backstage ALB URL shows "Looks like someone dropped the mic!" 404.

Root cause: Backstage config arrays replace rather than merge across config files. The production ConfigMap's extensions list overwrites the base config's page:catalog: path: / entry.

Fix (already applied): backstage/app-config.aws.yaml (rendered into the backstage-config ConfigMap by bootstrap.sh) now explicitly includes page:catalog: path: / alongside page:kubernetes: disabled: true.

Issue 2: Backstage catalog returns 401 / guest login broken

Symptom: All API calls return 401. Guest sign-in doesn't work.

Root cause: dangerouslyDisableDefaultAuthPolicy: true must be under backend.auth, not the top-level auth key.

Fix (already applied): Correct placement in backstage/app-config.aws.yaml: 

1
2
3
backend:
  auth:
    dangerouslyDisableDefaultAuthPolicy: true

Issue 3: Kubernetes tab shows 401 UNAUTHORIZED

Symptom: hello-service Kubernetes tab shows UNAUTHORIZED_ERROR, status code: 401 for all resources.

Root cause: K8S_SERVICE_ACCOUNT_TOKEN in Secrets Manager was REPLACE_ME — never populated with the real cluster token.

Fix (already applied): bootstrap.sh reads the token from backstage-sa-token secret and injects it into Secrets Manager automatically.

Issue 4: ClusterSecretStore fails — external-secrets-sa not found

Symptom: kubectl get clustersecretstore aws-secretsmanager shows InvalidProviderConfig. DORA exporter secret never syncs.

Root cause: ESO Helm chart creates a ServiceAccount named external-secrets, but the IAM trust policy and ClusterSecretStore both reference external-secrets-sa (which doesn't exist by default).

Fix (already applied): bootstrap.sh explicitly creates external-secrets-sa and annotates it with the IRSA role before applying the ClusterSecretStore.

Issue 5: DORA metrics missing from Grafana

Symptom: DORA Metrics dashboard shows "No data".

Root cause: dora-exporter.py was writing to CloudWatch only, but the Grafana dashboard queries Prometheus for dora_deploy_frequency_per_day.

Fix (already applied): Exporter now pushes to Pushgateway using the exact metric names the dashboard expects. CloudWatch publishing is retained as a secondary path.

Issue 6: hello-service ImagePullBackOff on EKS

Symptom: hello-service pods show ImagePullBackOff or no match for platform in manifest.

Root causes: - Image was in private GHCR (403 Forbidden) - Image built on Apple Silicon (arm64) but EKS nodes are amd64

Fix (already applied): helm-values-aws.yaml points to ECR. Build for the correct platform: 

1
2
3
4
cd services/hello-service
docker buildx build --platform linux/amd64 \
  -t "${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com/idp-mvp/hello-service:latest" \
  --push .

Issue 7: TechDocs 404 for hello-service

Symptom: hello-service Docs tab shows "no index.md in the root of the docs directory".

Root cause: dir:../../../services/hello-service relative traversal from a GitHub-URL-loaded catalog entity is unreliable.

Fix (already applied): backstage/catalog/services/hello-service/catalog-info.yaml uses an explicit GitHub URL: 

1
backstage.io/techdocs-ref: url:https://github.com/moatazeldebsy/backstage-platform-template/tree/main/services/hello-service

Issue 8: ArgoCD idp-services ApplicationSet missing

Symptom: Services don't appear in ArgoCD. platform-services app shows ComparisonError: app path does not exist.

Root cause: The idp-services ApplicationSet (which auto-discovers services/*/) was never applied.

Fix (already applied): bootstrap.sh Phase 4.6 applies aws/argocd/app-of-apps.yaml.

Issue 9: OpenCost and Pushgateway have no ALB ingress

Symptom: No public URL for OpenCost or Pushgateway after deployment.

This is intentional, not a fault. Both used to publish their own internet-facing ALB (~$16/mo each, no authentication in front of either). They are operator tools — OpenCost's data reaches users through the Backstage FinOps tab and the Grafana FinOps dashboard, and Pushgateway is written to by CI and scraped by Prometheus in-cluster. Reach the raw UIs with:

1
2
kubectl port-forward -n opencost   svc/opencost               9090:9090
kubectl port-forward -n monitoring svc/prometheus-pushgateway 9091:9091

The same applies to Prometheus, Alertmanager and the Argo Rollouts dashboard; bootstrap.sh prints every port-forward command in its closing banner.

Issue 10: Tech Radar shows empty / "not allowed to read"

Symptom: Tech Radar page is blank or shows a reading permission error.

Root cause: backend.reading.allow was missing from the ConfigMap, so Backstage blocked reads from raw.githubusercontent.com.

Fix (already applied): backstage/app-config.aws.yaml includes: 

1
2
3
4
5
backend:
  reading:
    allow:
      - host: raw.githubusercontent.com
      - host: github.com

Issue 11: Grafana login fails with multiple replicas

Symptom: Logging into Grafana succeeds but immediately redirects to login again.

Root cause: 2 Grafana replicas with ALB (no sticky sessions) — session created on pod A, next request hits pod B.

Fix (already applied): aws/observability/prometheus-stack-values.yaml sets replicas: 1.

Troubleshooting

For a comprehensive fresh-setup troubleshooting guide covering all AWS phases (Terraform, EKS, Backstage, ArgoCD, Crossplane, Observability), see docs/TROUBLESHOOTING.md.

Backstage pod not starting

1
2
kubectl logs -n backstage deployment/backstage --tail=50 | grep -E "error|Error|FATAL"
kubectl describe pod -n backstage -l app=backstage | grep -A10 "Events:"

Common causes: - Database not ready: check kubectl get pods -n backstage | grep postgres - Secret not synced: kubectl get externalsecret backstage-secrets -n backstage - Config YAML parse error: check logs for YAMLException

Stale Terraform lock

1
2
cd terraform
terraform force-unlock <lock-id>

EKS nodes not ready

1
2
3
kubectl get nodes
kubectl describe nodes | grep -A5 "Conditions:"
kubectl wait --for=condition=Ready node --all --timeout=600s

Re-seed QA metrics after Pushgateway restart

1
2
3
PUSHGATEWAY_URL=http://$(kubectl get ingress prometheus-pushgateway -n monitoring \
  -o jsonpath='{.status.loadBalancer.ingress[0].hostname}') \
  ./scripts/seed-qa-metrics.sh

Trigger DORA metrics collection manually

1
2
kubectl create job dora-manual-$(date +%s) --from=cronjob/dora-exporter -n monitoring
kubectl logs -n monitoring -l job-name=dora-manual-* --tail=20

Force Backstage catalog refresh

1
2
kubectl rollout restart deployment/backstage -n backstage
# Catalog repopulates within ~60 seconds

Cost Optimization

Free-tier reality

EKS is not free-tier compatible. Fixed monthly costs that cannot be eliminated while the cluster is running:

Resource Monthly floor Free tier?
EKS control plane ~$73 No
NAT Gateway ~$33 + data No
Secrets Manager (5 secrets) ~$2 No

Lowest cost approach: use ./scripts/bootstrap-local.sh --full for daily development. Deploy to AWS only for demos, then clean up immediately.

Default cost-optimized settings

terraform/terraform.tfvars.example, which setup.sh copies on first run, is tuned for the lowest spend that still actually runs the platform:

1
2
3
4
5
node_group_min_size      = 0        # scale-to-zero allowed (cost optimizer)
node_group_desired_size  = 6
node_group_max_size      = 8
enable_cost_optimizer    = true     # overnight Lambda scaler (see below)
budget_monthly_limit_usd = "100"    # SNS alert at $80 actual / $100 forecasted

This used to recommend desired_size = 1. A single t3.medium is 2 vCPU / 4 GiB, and rendering every chart and manifest this repo installs gives roughly 2.4 vCPU and 5.7 GiB of declared requests before per-node DaemonSet overhead (~0.3 vCPU and ~900 MiB each, for aws-node, kube-proxy, ebs-csi-node, promtail and the Datadog agent) — so one node could never have run it, and following that advice produced a cluster stuck in Pending.

Six is deliberately conservative rather than measured on a live cluster. Once the platform is up, check the real numbers before going lower — OpenCost and Prometheus are both deployed, and the Grafana FinOps dashboard reports per- namespace cost and efficiency. Note that within the t3 family the $/vCPU and $/GiB are identical, so only the node count changes the bill; moving to t3.large buys memory at exactly proportional cost.

Overnight node scaler

When enable_cost_optimizer = true, two EventBridge-triggered Lambdas run on schedule:

Lambda Schedule (UTC) Action
EKS node scaler 8 PM daily Scale node group → 0
RDS scheduler 8 PM daily Stop RDS instance
Both 7 AM daily Scale back to desired / start RDS

This saves ~60% of EC2 and RDS costs (assuming 9 h active / 24 h). To adjust the schedule:

1
2
3
# terraform/terraform.tfvars
cost_optimizer_scale_down_cron = "cron(0 22 * * ? *)"  # 10 PM UTC
cost_optimizer_scale_up_cron   = "cron(0 6  * * ? *)"  # 6 AM UTC

To disable entirely (always-on cluster): 

1
enable_cost_optimizer = false

Prometheus storage (already optimized)

aws/observability/prometheus-stack-values.yaml ships with cost-optimized defaults:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
prometheusSpec:
  retention: 3d            # time-based cap — reduce scrape history
  retentionSize: 15GB      # size-based cap — this is the load-bearing one
  storageSpec:
    volumeClaimTemplate:
      spec:
        storageClassName: gp3   # 20% cheaper than gp2
        resources:
          requests:
            storage: 20Gi       # headroom above retentionSize

Do not drop storage below retentionSize with room to spare, and do not remove retentionSize. retention alone is a time cap, so a busy cluster can fill the volume long before the window expires — a previous 5Gi volume filled in about four days and silently wedged ingestion. retentionSize is what actually bounds the volume; storage: 20Gi exists to give it headroom.

For longer history, raise both together (e.g. retention: 15d, retentionSize: 45GB, storage: 60Gi) and expect the EBS cost to rise with it.

Skip the AI/ML stack

KAgent + MLflow + MCP servers add ~6 platform ALBs plus one per MCP server per environment (~$36–72/month for the platform pieces alone, more once the MCP servers are promoted to staging and prod) and ~6 Gi EBS. Skip them unless needed:

1
2
3
4
5
6
7
# Core platform only (no AI/ML)
./scripts/bootstrap.sh

# Add AI/ML later when needed
./scripts/bootstrap-ai.sh --aws
# Remove it
./scripts/bootstrap-ai.sh --aws --destroy

Estimated monthly cost by configuration

Rough us-east-1 on-demand estimates at ~730 h/month. Treat them as order-of- magnitude only — the authoritative numbers are the AWS budget alert below and the OpenCost/Grafana FinOps dashboard, both of which this platform deploys.

The fixed floor is ~$135/mo before a single workload runs: EKS control plane (~$73), one NAT gateway (~$33), RDS db.t3.micro (~$17), plus EBS/S3/ECR/ Secrets (~$15). Nodes and ALBs are what actually move.

Config EC2 nodes Cost optimizer AI/ML ALBs Est. monthly
Default 6× t3.medium On (8 PM–7 AM UTC) Off 3 ~$280–310
Always-on, no AI 6× t3.medium Off Off 3 ~$370–400
Always-on + AI 6× t3.medium Off On 6 ~$430–490
Production per profiles/medium.tfvars Off Optional 6 ~$700–900

The ALB column understates a full AI deployment. Measured on the live cluster 2026-08-17, 27 ALB Ingresses existed with --adp and all three environments populated (dev/staging/prod each publish their own), not 6 — one per Ingress, since group.name is unused. Removing the six dead internal ones below brings that to 21. Treat the table as the core-platform baseline.

Scaffolded services no longer add to this by default: since #397 their helm-values-aws.yaml ships ingress.enabled: false, so a new service publishes no ALB unless you opt in. The 27 measured above predate that change and reflect environments whose services all had ingress enabled.

Where the money goes, and what has already been done about it:

  • Nodes dominate — 6× t3.medium is ~$182/mo on-demand. enable_cost_optimizer takes ~45% off that by scaling to zero overnight. Within the t3 family the $/vCPU and $/GiB are identical, so only the node count matters, not the size.
  • ALBs used to be the silent second-biggest line: every ALB Ingress provisions its own load balancer at ~$16/mo, and there were eleven. Prometheus, Alertmanager, Pushgateway, OpenCost and the Argo Rollouts dashboard no longer publish one — they are operator tools, reachable with kubectl port-forward (the bootstrap banner prints the exact commands). That is ~$82/mo, and it also took five unauthenticated endpoints off the public internet.
  • Six internal ALBs were pure waste and have been removed (2026-08-18). approval-service and the argocd/cost/github/incident/security MCP servers each published an internal ALB with a <service>.internal hostname. No private hosted zone exists, so those names resolved nowhere, and every real caller uses in-cluster Service DNS anyway — app-config.aws.yaml proxies straight to *.services-dev.svc.cluster.local. They routed nothing and billed ~$96/mo. Their ingress.enabled is now false, which is also more private than an internal ALB.
  • Consolidating the eight remaining internet-facing ALBs onto one would save ~$112/mo more. See Consolidating ALBs — it needs real DNS, so it is a project rather than a flag.

Consolidating ALBs

Eight services still publish their own internet-facing ALB at ~$16/mo each. They cannot share one today: every entry sets host: "" with path: /, so a shared alb.ingress.kubernetes.io/group.name produces colliding rules and the ALB controller rejects the group. Giving each a real hostname fixes that — the routing then keys on Host, not path, so no service has to move off / and no application code changes.

One ALB is not achievable while keeping internal services private. An IngressGroup maps to exactly one ALB with one scheme, so internal and internet-facing Ingresses cannot share a group. The realistic floor is one shared internet-facing ALB; anything that must stay private should publish no Ingress at all and be reached in-cluster, which is what the six removed above now do.

Steps, in order:

  1. Set domain_name in terraform/terraform.tfvars (a Route 53 zone must already exist for it). terraform/acm.tf is already gated on this and will provision a wildcard *.${domain_name} certificate with DNS validation — no new Terraform is needed for TLS.
  2. Give every internet-facing service a hostname in its helm-values-aws.yaml, e.g. host: hello-service.idp.example.com.
  3. Add a shared group to those same annotations: 
    1
2
3
4
    alb.ingress.kubernetes.io/group.name: idp-public
alb.ingress.kubernetes.io/group.order: "10"   # lower = evaluated first
alb.ingress.kubernetes.io/listen-ports: '[{"HTTP":80},{"HTTPS":443}]'
alb.ingress.kubernetes.io/certificate-arn: <wildcard cert ARN from acm.tf>
    Every Ingress carrying the same group.name merges onto one ALB.
  4. Point DNS at it — see below.

DNS: wildcard now, external-dns at scale

For this platform, start with a single wildcard ALIAS: one *.idp.example.com record to the shared ALB, matching the wildcard certificate acm.tf already issues. No controller, no extra IAM, and every future scaffolded service resolves the moment its Ingress joins the group. For a template repo that others adopt, that is the right default because it adds no prerequisites.

At organisation scale, the standard is external-dns over a delegated subdomain. The reason is lifecycle, not convenience: records are created and deleted with the Ingress, so a scaffolded service needs no ticket and no Terraform PR. If you adopt it:

  • Delegate, never grant apex access. NS-delegate dev.idp.example.com to a zone the platform owns. A controller with write access to the company apex zone is an unacceptable blast radius.
  • Scope the IRSA role to that hosted zone ARN only.
  • Set --txt-owner-id=<cluster-name> so several clusters can share a zone without overwriting each other's records. This is what makes it multi-cluster safe, and it is the setting most often missed.
  • Choose --policy deliberately. sync deletes records when the Ingress goes; upsert-only is safer on day one but leaks records forever, which becomes its own cleanup project.
  • Pair it with the wildcard cert or cert-manager. DNS automation without matching certificate automation just moves the ticket.

Multi-region caution: the V2 active-standby design (see multi-region.md) manages Route 53 health checks and failover records itself. external-dns with --policy=sync will fight hand-managed records in the same zone — give them separate zones, or be strict with --txt-owner-id.

Further cost optimisations (proposed, not yet implemented)

Measured against the real August 2026 bill, for a cluster that ran six days (2026-08-12 to 08-18) at $45–51/day:

1
2
3
4
5
6
  79.04  EKS control plane        (fixed, $0.10/hr per cluster)
  67.99  EC2 - Compute            (the nodes)
  56.26  Elastic Load Balancing   (one ALB per Ingress)
  29.02  VPC                      (NAT gateway)
  27.54  EC2 - Other              (EBS)
   5.28  RDS

Already implemented — single NAT gateway, Karpenter with a spot-first NodePool and spot-interruption handling, managed node group scaled to desired_size = 1 with Karpenter taking the burst, overnight scale-to-zero, 7-day EKS log retention, Prometheus storage limits, and eleven ALBs removed (five operator tools, then six dead internal ones).

Ranked by what is actually left:

# Proposal Est. saving Effort / risk
1 Destroy between sessions. Nothing else comes close — the platform costs ~$48/day running and ~$1.60/month destroyed. The state bucket and lock table are preserved precisely so a rebuild is cheap. ~$1,400/mo vs always-on None; already the practice
2 Consolidate the eight internet-facing ALBs onto one host-routed ALB ~$112/mo Medium — needs domain_name, DNS, ACM
3 Drop the NAT gateway in dev. ~$32/mo plus data processing, purely so private nodes can reach the internet. A dev cluster can run nodes in public subnets, or keep NAT and add S3/ECR/STS VPC endpoints so image pulls stop crossing it. $29–32/mo Medium; endpoints have their own hourly cost, so measure first
4 Force gp3 EBS. No volume_type is set anywhere in terraform/, so volumes land on the driver default. gp3 is ~20% cheaper than gp2 and decouples IOPS from size. ~$5/mo at current usage Low — a StorageClass change
5 Set retention on non-EKS log groups. Only eks.tf sets retention_in_days; the Lambda log group had none, meaning never expire. Small now, grows forever Low
6 Right-size the memory-optimized node group. It defaults to desired_size = 0 and only scales for AI workloads — confirm it returns to zero after bootstrap-ai.sh runs, rather than idling. Up to ~$60/mo if it idles Low — verification, not code
7 Avoid a second EKS control plane. At $73/mo each, the V2 multi-region active-standby design doubles this line before a single workload runs. Worth confirming the standby needs a live control plane rather than being rebuilt on failover. $73/mo Design decision

Not ours, but the largest residual line in this account once the platform is destroyed: a 50 GB RDS snapshot (infra-landscape-dev-postgres-final-snapshot, ~$4.75/mo) and a disabled KMS key (alias/mokey, created 2021, ~$1/mo) — both left over from an unrelated project. Together they cost roughly 3.5× everything this platform leaves behind.

Budget alert

A budget is provisioned in Terraform at $100/month with SNS alerts at: - 80% actual (~$80 spent) — early warning - 100% forecasted — projected overage

Set budget_alert_email in terraform/terraform.tfvars to receive email alerts directly. The SNS → Slack Lambda bridge fires if idp-mvp/slack-webhook is populated in Secrets Manager.

Production Hardening

Scale EKS for production load

1
2
3
4
5
6
# terraform/terraform.tfvars
node_instance_types     = ["t3.large"]
node_group_min_size     = 2
node_group_desired_size = 3
node_group_max_size     = 8
enable_cost_optimizer   = false   # keep nodes up 24/7 for production

Enable RDS deletion protection and backups

1
2
3
4
5
# terraform/terraform.tfvars
environment             = "prod"   # enables deletion_protection=true, skip_final_snapshot=false
# Note: backup_retention_period is currently set to 1 (cost-optimized default).
# For production, increase it in terraform/rds.tf:
#   backup_retention_period = 7

Enable ALB sticky sessions (if scaling Grafana)

Add to Grafana ALB ingress annotations: 

1
alb.ingress.kubernetes.io/target-group-attributes: stickiness.enabled=true,stickiness.lb_cookie.duration_seconds=86400

Rotate GitHub token

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
aws secretsmanager get-secret-value --secret-id idp-mvp/backstage --region us-east-1 \
  --query SecretString --output text | python3 -c "
import sys, json
d = json.load(sys.stdin)
d['GITHUB_TOKEN'] = 'ghp_NEW_TOKEN_HERE'
print(json.dumps(d))
" | aws secretsmanager put-secret-value \
  --secret-id idp-mvp/backstage --region us-east-1 --secret-string file:///dev/stdin

kubectl annotate externalsecret backstage-secrets -n backstage force-sync=$(date +%s) --overwrite
kubectl rollout restart deployment/backstage -n backstage

Set PagerDuty / Jira credentials (optional)

Both are optional — left unset, the On-Call and Issues tabs render their empty state rather than erroring, exactly as they do locally. Supply them either through Terraform:

1
2
3
4
# terraform.tfvars
pagerduty_token = "your-read-only-rest-api-key"
jira_token      = "base64-of-email:api_token"
jira_url        = "https://your-company.atlassian.net"

...or by patching the existing secret in place, which avoids a Terraform apply:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
aws secretsmanager get-secret-value --secret-id idp-mvp/backstage --region us-east-1 \
  --query SecretString --output text | python3 -c "
import sys, json
d = json.load(sys.stdin)
d['PAGERDUTY_TOKEN'] = 'YOUR_PAGERDUTY_KEY'
d['JIRA_TOKEN']      = 'YOUR_BASE64_EMAIL_COLON_TOKEN'
d['JIRA_URL']        = 'https://your-company.atlassian.net'
print(json.dumps(d))
" | aws secretsmanager put-secret-value \
  --secret-id idp-mvp/backstage --region us-east-1 --secret-string file:///dev/stdin

kubectl annotate externalsecret backstage-secrets -n backstage force-sync=$(date +%s) --overwrite
kubectl rollout restart deployment/backstage -n backstage

JIRA_TOKEN is base64(email:api_token), not the raw API token. Generate the token at https://id.atlassian.com/manage-profile/security/api-tokens, then printf 'you@example.com:TOKEN' | base64.

Leaving JIRA_URL empty keeps app-config.aws.yaml on its RFC 2606 https://jira.invalid default, which never resolves — deliberate, so an unconfigured proxy cannot send requests to a domain nobody here controls.

Pod Disruption Budgets

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
kubectl apply -f - <<EOF
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: backstage-pdb
  namespace: backstage
spec:
  minAvailable: 1
  selector:
    matchLabels:
      app: backstage
EOF

Cleanup & Destroy

1
2
# Safe teardown — runs all phases in the correct order
./scripts/cleanup.sh --cluster-name idp-mvp --force

cleanup.sh runs nine ordered phases (0–8). The order is the whole point — see why terraform destroy alone is not enough.

Phase What it does
0 — Stop the writers Deletes ArgoCD ApplicationSets, then workload Applications with cascade, so the load balancer controller tears down its own ALBs instead of leaking them. Also scales the Loki write path to zero — Loki is Helm-installed, not ArgoCD-managed, and will otherwise keep writing into the S3 bucket Phase 5 is trying to empty.
1 — ALBs Deletes any orphaned Kubernetes-managed ALBs (k8s-*) and stale k8s-* security groups. Both block VPC deletion, and the security groups only release asynchronously, so this retries.
2 — RDS protection Disables deletion protection on the Backstage RDS instance
3 — Scaffolded services Auto-discovers user-scaffolded services in services/ (excludes built-ins). Deletes their ArgoCD Applications, uninstalls Helm releases, removes services/<name>/ and commits the deletion. Runs while EKS is still up so ArgoCD can cascade cleanly.
4 — Crossplane resources Finds everything tagged idp:provisioner=crossplane via the Resource Groups Tagging API and deletes it: S3 buckets (all versions + delete markers), RDS instances, DynamoDB tables, SQS queues. MSK topics go with the cluster in Phase 6.
5 — S3 + ECR empty Empties Terraform-managed S3 buckets and force-deletes <cluster>-prefixed ECR repos. AWS blocks bucket deletion when non-empty and force_destroy = false, which is the case for every bucket here.
6 — Terraform destroy Destroys all Terraform-managed resources: EKS, VPC, IAM/IRSA, RDS, ECR, KMS, Secrets Manager. Retried, because BucketNotEmpty and security-group DependencyViolation are both expected and self-healing.
7 — CloudWatch Deletes log groups EKS and Lambda create at runtime (/aws/eks/<cluster>, /aws/containerinsights/<cluster>, /aws/lambda/<cluster>-*) — these survive the cluster and keep billing
8 — Verify Checks EKS, RDS, ALBs, Crossplane-tagged resources and log groups are all zero

Preserved deliberately: the Terraform state bucket (<cluster>-terraform-state-*), its DynamoDB lock table (<cluster>-terraform-locks), and any ECR repo whose name does not contain the cluster name. Those bare-named repos were never Terraform-managed, so a teardown leaves them — see the ownership table below.

What Terraform owns, and what it cannot destroy

No, terraform destroy on its own will not clean up this platform, and the gap is not an oversight — it follows from how the platform is built. Terraform owns the account and cluster foundation. Controllers running inside the cluster create AWS resources of their own in response to Kubernetes objects, and Terraform has never heard of those.

Resource Owner Why
EKS, VPC, subnets, NAT, IAM/IRSA, OIDC provider, RDS, KMS, Secrets Manager, budgets, ACM Terraform (terraform/, ~270 resources) Account/cluster foundation, applied once by bootstrap.sh
ECR repos named <cluster>/<service> Terraform (force_delete = true) Declared in ecr.tf; images are removed with the repo
ALBs, target groups, k8s-* security groups AWS Load Balancer Controller Created at runtime from Ingress objects. They exist in no state file, and they block VPC deletion. Delete the Ingress and the controller cleans up after itself; delete the cluster first and they orphan at ~$16/mo each, invisible to Terraform. This is why Phase 0 runs before Phase 6.
S3/RDS/DynamoDB/SQS/MSK from Claims Crossplane Provisioned from Claims committed to Git. Compositions set deletionPolicy: Orphan on purpose — a deleted Claim must not silently destroy a team's data. They are found by the idp:provisioner=crossplane tag instead.
ECR repos created by bootstrap-ai.sh The script, imperatively AI/MCP service repos are created on demand, not declared in Terraform
Contents of S3 buckets and ECR repos Nobody — runtime data Terraform can delete a bucket but AWS refuses while it holds objects and force_destroy = false. Contents must be emptied first, which is Phase 5.
/aws/eks/*, /aws/lambda/* log groups EKS / Lambda, at runtime Created by the services themselves and outlive the cluster
Kubeconfig contexts Local machine Not an AWS resource; prune with kubectl config delete-context

Two more reasons ordering matters, both learned the hard way:

  • KMS keys and Secrets Manager secrets are never deleted immediately. Terraform "destroys" them, but AWS only schedules deletion — 7–30 days for a KMS key, 7–30 for a secret. They keep billing (~$1/mo and ~$0.40/mo) until the window closes. This is AWS behaviour, not a bug in the teardown.
  • Once EKS is gone, Terraform cannot even plan. The kubernetes, helm and kubectl providers in main.tf take their host from module.eks.cluster_endpoint. With the cluster destroyed that value is empty and the provider fails to configure, so a destroy interrupted after the cluster goes but before the VPC does cannot simply be re-run. cleanup.sh detects this and falls back to per-resource -target destroys.

If a teardown stalls, check the object count of the bucket being emptied rather than the log — _empty_versioned_bucket only logs when it finishes, so a slow phase and a wedged one look identical from the output.

Note on Crossplane resources: Because all Compositions use deletionPolicy: Orphan, Crossplane resources survive Claim deletion. Phase 3 of cleanup.sh handles them automatically by querying the idp:provisioner=crossplane tag. RDS deletions are asynchronous — the script initiates them and notes it in the summary.

To tear down only the AI/ML components on AWS without touching the core platform:

1
./scripts/bootstrap-ai.sh --aws --destroy

Important — local vs AWS separation: bootstrap-ai.sh --destroy scopes deletions to the active environment. Passing --aws removes aws/ ingresses and the MLflow AWS overlay; omitting it removes only local/ ingresses. Never run bootstrap-ai.sh --destroy (without --aws) while KUBECONFIG points at EKS — it will target the wrong cluster.

To tear down only the local Kind/Rancher cluster (no AWS resources are affected):

1
./scripts/bootstrap-local.sh --destroy

This runs the same scaffolded-service cleanup logic (ArgoCD Applications deleted, Helm releases uninstalled, services/<name>/ directories removed and committed) before the Kind cluster or Rancher Desktop namespaces are deleted.