Skip to content

IDP Scaling Runbook

This runbook covers when and how to move between the three reference tiers as your organisation grows, what to apply at each step, and how to verify the outcome.

Tier Reference

Tier Teams Engineers EKS nodes (desired) RDS class Search backend ArgoCD controllers
Small ≤ 25 ≤ 150 3 (t3.large) db.t3.medium Lunr (in-mem) 1
Medium 26–75 150–750 6 (m5.xlarge) db.m5.large Elasticsearch 1 (scaled)
Large 75+ 750+ 12 (m5.2xlarge) db.r5.xlarge Elasticsearch 3 (sharded)

Signals to move up a tier

Move from Small → Medium when any of these are true: - Backstage catalog entity count > 500 - Backstage pod CPU > 70% sustained for > 10 minutes - ArgoCD sync queue depth > 20 apps waiting - Team count approaches 20 (plan ahead — provisioning takes 30–60 min)

Move from Medium → Large when any of these are true: - Team count approaches 60 - ArgoCD application count > 250 - EKS node group desired_size > 15 consistently - RDS CPU > 60% sustained (Backstage catalog queries)

Step 1 — Apply Terraform profile (infrastructure resize)

1
2
3
4
5
6
7
cd terraform

# Dry-run first — inspect what changes
terraform plan -var-file=profiles/medium.tfvars

# Apply (EKS node group resize is rolling, ~10 min; RDS Multi-AZ failover takes ~5 min)
terraform apply -var-file=profiles/medium.tfvars

Key changes per tier: - Small → Medium: EKS nodes t3.large→m5.xlarge, RDS db.t3.medium→db.m5.large, rds_multi_az=true (triggers RDS modification, brief failover), Karpenter enabled. - Medium → Large: EKS nodes m5.xlarge→m5.2xlarge, RDS db.m5.large→db.r5.xlarge, VPC CIDR expands (requires VPC replacement — plan a maintenance window).

Large tier VPC note: Expanding vpc_cidr from 10.0.0.0/16 to 10.0.0.0/8 requires destroying and recreating the VPC and all dependent resources. Do this with a blue/green cluster approach — provision the new VPC in a separate Terraform workspace, migrate teams, then decommission the old one.

Step 2 — Apply Helm tier values (platform service resize)

Backstage

1
2
3
4
5
# Replace <tier> with small, medium, or large
helm upgrade backstage backstage/backstage \
  -n backstage \
  -f helm/values-tiers/backstage-<tier>.yaml \
  --reuse-values

At Medium+, Elasticsearch must be running before applying the medium/large values:

1
2
3
4
5
6
7
8
9
# Deploy Elasticsearch (one-time)
helm upgrade --install elasticsearch elastic/elasticsearch \
  -n search --create-namespace \
  --set replicas=1 \
  --set resources.requests.memory=2Gi

# Then install the ES search backend plugin in Backstage
# See backstage/app/packages/backend/src/index.ts — add:
# backend.add(import('@backstage/plugin-search-backend-module-elasticsearch'));

ArgoCD

1
2
3
4
helm upgrade argocd argo/argo-cd \
  -n argocd \
  -f helm/values-tiers/argocd-<tier>.yaml \
  --reuse-values

At Large, controller sharding is enabled (3 replicas with round-robin algorithm). Verify sharding is active:

1
2
argocd admin controller-info
# Should show shard assignments across 3 controller pods

Step 3 — Onboard teams at scale (Team Namespace template)

Each new team should go through the Provision Team Namespace Backstage template. It creates in one scaffold run: - team-<slug> Namespace with tier label and cost tags - ResourceQuota (Small/Medium) or LimitRange defaults (Large) - ArgoCD AppProject scoped to the namespace - idp-developer RoleBinding for team members - deployer ServiceAccount for CI/CD - Backstage Group entity (auto-registered in catalog)

Navigate to: Backstage → Create → Provision Team Namespace

For bulk onboarding (migration from existing namespaces), use the scaffold API directly:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
# Trigger scaffold via Backstage API for each team
curl -X POST https://<backstage-url>/api/scaffolder/v2/tasks \
  -H "Authorization: Bearer $BACKSTAGE_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "templateRef": "template:default/team-namespace",
    "values": {
      "teamName": "payments",
      "ownerGroup": "engineering",
      "costCenter": "CC-1234",
      "tier": "medium",
      "githubOrg": "your-org",
      "platformRepo": "backstage-platform-template"
    }
  }'

Step 4 — Verify after tier change

Infrastructure

1
2
3
4
5
6
# EKS nodes are the right instance type and count
kubectl get nodes -o wide

# RDS Multi-AZ is active (Medium+)
aws rds describe-db-instances \
  --query 'DBInstances[].{id:DBInstanceIdentifier,multiAZ:MultiAZ,class:DBInstanceClass}'

Backstage

1
2
3
4
5
# Pods are healthy and replicas match tier
kubectl get pods -n backstage

# Search backend is reachable (Medium+)
kubectl logs -n backstage deploy/backstage | grep -i elasticsearch

ArgoCD

1
2
3
4
5
# All apps are synced; no apps stuck in queue
argocd app list | grep -v Synced

# Shard distribution (Large)
argocd admin controller-info

Quota policy

1
2
3
4
5
6
# Kyverno generated the quota for a test namespace
kubectl get resourcequota -n team-<slug>

# Attempt a quota-busting pod (should be denied at Medium/Small)
kubectl run quota-test --image=nginx -n team-<slug> \
  --overrides='{"spec":{"containers":[{"name":"c","image":"nginx","resources":{"requests":{"cpu":"100"}}}]}}'

Scaling checklist — implementation status

The items below map to the original 25-team scaling analysis. ✅ = shipped in v0.4.0.

Area Item Status
Tenancy Per-team namespace (quota, LimitRange, NetworkPolicy) ✅ team-namespace scaffold skeleton
Tenancy services-dev/staging/prod ResourceQuota + LimitRange ✅ kubernetes/namespaces/services-quota.yaml
Tenancy Pod Security Admission per namespace ✅ labels on all namespaces
Ownership Per-team ArgoCD AppProject ✅ skeleton/argocd-project.yaml
Ownership Per-team ArgoCD ApplicationSet (teams/<team>/services/*) ✅ skeleton/applicationset.yaml
Ownership idp-developer / idp-team-lead / idp-platform-admin ClusterRoles ✅ kubernetes/rbac/cluster-roles.yaml
Secrets Per-team namespace-scoped SecretStore ✅ skeleton/secret-store.yaml + terraform/iam-team-secret-store.tf
Secrets Per-team IRSA roles scoped to /<team>/* ✅ terraform/iam-team-secret-store.tf
Crossplane Kyverno auto-inject idp:team tag on all claims ✅ kubernetes/policies/kyverno/crossplane-team-label-policy.yaml
Crossplane Validate: block claims with no owner/costCenter ✅ same policy (Enforce mode)
Crossplane team field on all 5 XRDs + compositions ✅ S3, RDS, DynamoDB, SQS, Kafka
CI/CD Concurrency groups + max-parallel cap ✅ all workflows
CI/CD GitHub App replacing PAT ✅ auto-merge-onboarding.yml + app-config.aws.yaml
Backstage Small/Medium/Large Helm value tiers ✅ helm/values-tiers/
Backstage GitHub org sync (User/Group auto-import) ✅ app-config.aws.yaml githubOrg provider
Backstage Catalog consolidation (49 URLs → 1 file) ✅ backstage/catalog/all-templates.yaml
Backstage GitHub discovery for scaffolded services (replaces the URL catalog list) ✅ catalog.providers.github.idpOrg in app-config.yaml / app-config.aws.yaml — topic-filtered, 15-min sync
Backstage Template versioning (v1 + blessed/advanced tags) ✅ all 64 templates
Backstage Permission framework (permission.enabled: true) ✅ config added; backend plugin code still needed
Observability DORA team= label on all Prometheus metrics ✅ dora-exporter.py
Observability TEAM_MAP env var for repo→team mapping ✅ dora-exporter.py + CronJob YAMLs
Observability Per-team Grafana folder via sidecar ConfigMap ✅ skeleton/grafana-folder.yaml + sidecar enabled
Observability Kyverno install in bootstrap scripts ✅ Step 9b / Phase 3.8

Still open (not yet implemented)

Item Notes
Group-based template visibility in Backstage Requires permission backend plugin code in backstage/app/packages/backend/
Per-team Grafana RBAC (org → team mapping) Grafana folder exists; team RBAC provisioning via API not yet automated
DORA team dimension in Grafana dashboard Dashboard JSON needs team variable + per-team panel
Thanos / Mimir for Prometheus HA Plan when > 1M active series
CODEOWNERS auto-generation in scaffold templates Add CODEOWNERS file step in service template steps:

Multi-cluster topology (Large tier)

At Large scale, split into two clusters to contain blast radius:

Cluster Contains
idp-platform Backstage, ArgoCD, MCP servers, observability, Kyverno
idp-workloads All team-* namespaces, Crossplane, ESO

Steps: 1. Run terraform apply -var-file=profiles/large.tfvars with cluster_name=idp-workloads in a separate Terraform workspace. 2. Register the workload cluster in ArgoCD on the platform cluster: 

1
argocd cluster add <workload-kubeconfig-context>
3. Update ArgoCD ApplicationSets to target https://<workload-cluster-api> for team apps. 4. Update Kyverno ClusterPolicy to target the workload cluster (install Kyverno on workload cluster).

Rollback

All changes are backward-compatible. To roll back a tier:

1
2
3
4
5
6
# Terraform: reapply the previous tier profile
terraform apply -var-file=profiles/small.tfvars

# Helm: re-apply smaller tier values
helm upgrade backstage backstage/backstage -n backstage \
  -f helm/values-tiers/backstage-small.yaml --reuse-values

Note: RDS instance class downgrades require a brief DB restart. EKS node group downgrades are rolling and non-disruptive if pods fit on the smaller nodes.