IDP Scaling Runbook¶

This runbook covers when and how to move between the three reference tiers as your organisation grows, what to apply at each step, and how to verify the outcome.

Tier Reference¶

Tier	Teams	Engineers	EKS nodes (desired)	RDS class	Search backend	ArgoCD controllers
Small	≤ 25	≤ 150	3 (t3.large)	db.t3.medium	Lunr (in-mem)	1
Medium	26–75	150–750	6 (m5.xlarge)	db.m5.large	Elasticsearch	1 (scaled)
Large	75+	750+	12 (m5.2xlarge)	db.r5.xlarge	Elasticsearch	3 (sharded)

Signals to move up a tier¶

Move from Small → Medium when any of these are true: - Backstage catalog entity count > 500 - Backstage pod CPU > 70% sustained for > 10 minutes - ArgoCD sync queue depth > 20 apps waiting - Team count approaches 20 (plan ahead — provisioning takes 30–60 min)

Move from Medium → Large when any of these are true: - Team count approaches 60 - ArgoCD application count > 250 - EKS node group desired_size > 15 consistently - RDS CPU > 60% sustained (Backstage catalog queries)

Step 1 — Apply Terraform profile (infrastructure resize)¶

Preferred: ./scripts/bootstrap.sh --profile medium applies the profile and remembers it, so later bootstrap runs don't silently fall back to terraform.tfvars. To inspect the change by hand first:

cd terraform

# Dry-run first — inspect what changes
terraform plan -var-file=profiles/medium.tfvars

# Apply (EKS node group resize is rolling, ~10 min; RDS Multi-AZ failover takes ~5 min)
terraform apply -var-file=profiles/medium.tfvars

Key changes per tier: - Small → Medium: EKS nodes t3.large→m5.xlarge, RDS db.t3.medium→db.m5.large, rds_multi_az=true (triggers RDS modification, brief failover), Karpenter enabled. - Medium → Large: EKS nodes m5.xlarge→m5.2xlarge, RDS db.m5.large→db.r5.xlarge, platform node group grows to the profile's desired size (Karpenter still takes service burst).

Large tier VPC note: all tiers use vpc_cidr = 10.0.0.0/16 — the largest CIDR AWS accepts on a VPC (/16–/28), so there is no VPC replacement between tiers. If pod IPs run short at this scale, add a secondary CIDR block (e.g. 100.64.0.0/16) with VPC-CNI custom networking and enable prefix delegation, rather than widening the primary CIDR.

Step 2 — Apply Helm tier values (platform service resize)¶

Backstage¶

# Replace <tier> with small, medium, or large
helm upgrade backstage backstage/backstage \
  -n backstage \
  -f helm/values-tiers/backstage-<tier>.yaml \
  --reuse-values

At Medium+, Elasticsearch must be running before applying the medium/large values:

# Deploy Elasticsearch (one-time)
helm upgrade --install elasticsearch elastic/elasticsearch \
  -n search --create-namespace \
  --set replicas=1 \
  --set resources.requests.memory=2Gi

# Then install the ES search backend plugin in Backstage
# See backstage/app/packages/backend/src/index.ts — add:
# backend.add(import('@backstage/plugin-search-backend-module-elasticsearch'));

ArgoCD¶

helm upgrade argocd argo/argo-cd \
  -n argocd \
  -f helm/values-tiers/argocd-<tier>.yaml \
  --reuse-values

At Large, controller sharding is enabled (3 replicas with round-robin algorithm). Verify sharding is active:

argocd admin controller-info
# Should show shard assignments across 3 controller pods

Step 3 — Onboard teams at scale (Team Namespace template)¶

Each new team should go through the Provision Team Namespace Backstage template. It creates in one scaffold run: - team-<slug> Namespace with tier label and cost tags - ResourceQuota (Small/Medium) or LimitRange defaults (Large) - ArgoCD AppProject scoped to the namespace - idp-developer RoleBinding for team members - deployer ServiceAccount for CI/CD - Backstage Group entity (auto-registered in catalog)

Navigate to: Backstage → Create → Provision Team Namespace

For bulk onboarding (migration from existing namespaces), use the scaffold API directly:

# Trigger scaffold via Backstage API for each team
curl -X POST https://<backstage-url>/api/scaffolder/v2/tasks \
  -H "Authorization: Bearer $BACKSTAGE_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "templateRef": "template:default/team-namespace",
    "values": {
      "teamName": "payments",
      "ownerGroup": "engineering",
      "costCenter": "CC-1234",
      "tier": "medium",
      "githubOrg": "your-org",
      "platformRepo": "backstage-platform-template"
    }
  }'

Step 4 — Verify after tier change¶

Infrastructure¶

# EKS nodes are the right instance type and count
kubectl get nodes -o wide

# RDS Multi-AZ is active (Medium+)
aws rds describe-db-instances \
  --query 'DBInstances[].{id:DBInstanceIdentifier,multiAZ:MultiAZ,class:DBInstanceClass}'

Backstage¶

# Pods are healthy and replicas match tier
kubectl get pods -n backstage

# Search backend is reachable (Medium+)
kubectl logs -n backstage deploy/backstage | grep -i elasticsearch

ArgoCD¶

# All apps are synced; no apps stuck in queue
argocd app list | grep -v Synced

# Shard distribution (Large)
argocd admin controller-info

Quota policy¶

# Kyverno generated the quota for a test namespace
kubectl get resourcequota -n team-<slug>

# Attempt a quota-busting pod (should be denied at Medium/Small)
kubectl run quota-test --image=nginx -n team-<slug> \
  --overrides='{"spec":{"containers":[{"name":"c","image":"nginx","resources":{"requests":{"cpu":"100"}}}]}}'

Scaling checklist — implementation status¶

The items below map to the original 25-team scaling analysis. ✅ = shipped in v0.4.0.

Area	Item	Status
Tenancy	Per-team namespace (quota, LimitRange, NetworkPolicy)	✅ team-namespace scaffold skeleton
Tenancy	`services-dev/staging/prod` ResourceQuota + LimitRange	✅ `kubernetes/namespaces/services-quota.yaml`
Tenancy	Pod Security Admission per namespace	✅ labels on all namespaces
Ownership	Per-team ArgoCD AppProject	✅ `skeleton/argocd-project.yaml`
Ownership	Per-team ArgoCD ApplicationSet (`teams/<team>/services/*`)	✅ `skeleton/applicationset.yaml`
Ownership	`idp-developer` / `idp-team-lead` / `idp-platform-admin` ClusterRoles	✅ `kubernetes/rbac/cluster-roles.yaml`
Secrets	Per-team namespace-scoped SecretStore	✅ `skeleton/secret-store.yaml` + `terraform/iam-team-secret-store.tf`
Secrets	Per-team IRSA roles scoped to `/<team>/*`	✅ `terraform/iam-team-secret-store.tf`
Crossplane	Kyverno auto-inject `idp:team` tag on all claims	✅ `kubernetes/policies/kyverno/crossplane-team-label-policy.yaml`
Crossplane	Validate: block claims with no owner/costCenter	✅ same policy (Enforce mode)
Crossplane	`team` field on all 5 XRDs + compositions	✅ S3, RDS, DynamoDB, SQS, Kafka
CI/CD	Concurrency groups + `max-parallel` cap	✅ all workflows
CI/CD	GitHub App replacing PAT	✅ `auto-merge-onboarding.yml` + `app-config.aws.yaml`
Backstage	Small/Medium/Large Helm value tiers	✅ `helm/values-tiers/`
Backstage	GitHub org sync (User/Group auto-import)	✅ `app-config.aws.yaml` `githubOrg` provider
Backstage	Catalog consolidation (49 URLs → 1 file)	✅ `backstage/catalog/all-templates.yaml`
Backstage	GitHub discovery for scaffolded services (replaces the URL catalog list)	✅ `catalog.providers.github.idpOrg` in `app-config.yaml` / `app-config.aws.yaml` — topic-filtered, 15-min sync
Backstage	Template versioning (`v1` + `blessed`/`advanced` tags)	✅ all 59 active templates
Backstage	Permission framework (`permission.enabled: true`)	✅ config added; backend plugin code still needed
Observability	DORA `team=` label on all Prometheus metrics	✅ `dora-exporter.py`
Observability	`TEAM_MAP` env var for repo→team mapping	✅ `dora-exporter.py` + CronJob YAMLs
Observability	Per-team Grafana folder via sidecar ConfigMap	✅ `skeleton/grafana-folder.yaml` + sidecar enabled
Observability	Kyverno install in bootstrap scripts	✅ Step 9b / Phase 3.8

Still open (not yet implemented)¶

Item	Notes
Group-based template visibility in Backstage	Requires permission backend plugin code in `backstage/app/packages/backend/`
Per-team Grafana RBAC (org → team mapping)	Grafana folder exists; team RBAC provisioning via API not yet automated
DORA team dimension in Grafana dashboard	Dashboard JSON needs `team` variable + per-team panel
Thanos / Mimir for Prometheus HA	Plan when > 1M active series
CODEOWNERS auto-generation in scaffold templates	Add `CODEOWNERS` file step in service template `steps:`

Multi-cluster topology (Large tier)¶

At Large scale, split into two clusters to contain blast radius:

Cluster	Contains
`idp-platform`	Backstage, ArgoCD, MCP servers, observability, Kyverno
`idp-workloads`	All `team-*` namespaces, Crossplane, ESO

Steps: 1. Run terraform apply -var-file=profiles/large.tfvars with cluster_name=idp-workloads in a separate Terraform workspace. 2. Register the workload cluster in ArgoCD on the platform cluster:

argocd cluster add <workload-kubeconfig-context>

3. Update ArgoCD ApplicationSets to target https://<workload-cluster-api> for team apps. 4. Update Kyverno ClusterPolicy to target the workload cluster (install Kyverno on workload cluster).

Rollback¶

All changes are backward-compatible. To roll back a tier:

# Terraform: reapply the previous tier profile
terraform apply -var-file=profiles/small.tfvars

# Helm: re-apply smaller tier values
helm upgrade backstage backstage/backstage -n backstage \
  -f helm/values-tiers/backstage-small.yaml --reuse-values

Note: RDS instance class downgrades require a brief DB restart. EKS node group downgrades are rolling and non-disruptive if pods fit on the smaller nodes.