Team Management¶

End-to-end guide for onboarding a new engineering team onto the IDP — from provisioning their isolated namespace to giving them a self-service Grafana folder and a scoped ArgoCD ApplicationSet.

Overview¶

Each team gets:

Resource	What it provides
`team-<name>` Kubernetes namespace	Isolated workload space with quota, LimitRange, NetworkPolicy
ArgoCD `AppProject`	Scoped sync: only that namespace, only that team's repos
ArgoCD `ApplicationSet`	Auto-discovers services under `teams/<name>/services/*`
Kyverno auto-quota	Quota generated from `idp.io/tier` namespace label
Namespace-scoped `SecretStore`	Access only to `/<name>/*` in AWS Secrets Manager (IRSA)
`idp-developer` RoleBinding	Team members can deploy + observe in their namespace
Grafana folder	Per-team dashboard folder via sidecar ConfigMap

Step 1 — Provision the team namespace via Backstage¶

Open the Backstage catalog → Create → search "Provision Team Namespace" (tagged blessed).

Fill in:

Field	Example	Notes
Team name	`payments`	Lowercase alphanum + hyphens, 3–30 chars
Owner group	`group:default/platform-team`	Existing Backstage Group this new team nests under
Cost center	`CC-1234`	Applied to all Crossplane resources
Initial team members	`alice, bob`	Optional — GitHub logins to add to the new GitHub team immediately
Sizing tier	`small`	Small ≤25 teams / Medium 26–75 / Large 75+
GitHub organisation (platform repo)	`moatazeldebsy`	Pre-filled — owner of the platform repo the PR opens against
GitHub organisation (teams)	`platform-demo-idp`	Pre-filled — the GitHub Org the new team is actually created in; must match `catalog.providers.githubOrg` or the Group won't sync
Platform repo	`backstage-platform-template`	Pre-filled
AWS region	`us-east-1`	Must match cluster region
ESO IAM role ARN	(see Step 2)	Leave blank to skip SecretStore

Click Create. Unlike the k8s manifests below, the GitHub team itself is created immediately (via the GitHub API, idempotent — reruns reuse an existing team) — it does not wait for a PR merge. A PR is then opened on the platform repo at team/payments-namespace for the Kubernetes-side resources only; no catalog YAML is part of that PR (see Step 3).

Step 2 — Create the per-team IRSA role (AWS only)¶

Before merging the scaffold PR, provision the IAM role that allows the team's External Secrets Operator ServiceAccount to read secrets under /payments/*:

cd terraform
terraform apply -var='team_eso_roles=[{name="payments",cost_center="CC-1234"}]'

# Get the role ARN output and paste it into the Backstage scaffold form
terraform output team_eso_role_arns
# → { "payments" = "arn:aws:iam::123456789012:role/team-payments-eso" }

Re-run the scaffold with the ARN in the ESO IAM role ARN field, or patch the secret-store.yaml in the PR directly.

For local (Kind) development, skip this step — the SecretStore is omitted when iamRoleArn is blank.

Step 3 — Merge the scaffold PR¶

Merge team/payments-namespace on the platform repo. ArgoCD's idp-teams ApplicationSet (aws/argocd/app-of-apps.yaml, local/argocd/app-of-apps-local.yaml) picks up the new kubernetes/teams/payments/ directory as Application team-payments and syncs it within its git polling interval (about 3 minutes), creating:

kubernetes/teams/payments/
├── namespace.yaml          # team-payments namespace + labels
├── resource-quota.yaml     # tier-based ResourceQuota
├── limit-range.yaml        # default container requests/limits
├── network-policy.yaml     # deny-all ingress except intra-ns + monitoring
├── rbac.yaml               # deployer SA + idp-developer RoleBinding
├── argocd-project.yaml     # AppProject scoped to team-payments
├── applicationset.yaml     # scans teams/payments/services/*
├── secret-store.yaml       # namespace-scoped SecretStore (if IAM role set)
└── grafana-folder.yaml     # ConfigMap → Grafana sidecar creates folder

ArgoCD keeps these in sync with git (manual kubectl edits are reverted), but it does not prune: deleting the directory from git leaves the namespace and its workloads in place. Removing a team is a deliberate manual step:

kubectl delete application team-payments -n argocd
kubectl delete namespace team-payments

The Backstage Group entity for the team is not part of this PR. Teams and their members are synced live from the GitHub Org (catalog.providers.githubOrg, every 30 min on AWS) — the Group appears automatically once the GitHub team created in Step 1 is picked up by the next sync (trigger a manual catalog refresh to see it sooner). See ADR-0004.

Step 4 — Add services for the team¶

Team services follow the teams/<teamName>/services/<serviceName>/ path convention:

teams/
└── payments/
    └── services/
        ├── orders-api/
        │   ├── helm-values-aws.yaml
        │   └── helm-values-local.yaml
        └── notifications-worker/
            ├── helm-values-aws.yaml
            └── helm-values-local.yaml

Important: Do not put team service values under services/<teamName>/. services/* is reserved for legacy platform-owned services and is scanned by the global ApplicationSet. Team dirs there would create broken ArgoCD Applications.

The per-team ApplicationSet detects new service directories automatically and creates ArgoCD Applications scoped to team-payments namespace + AppProject.

Creating a service via scaffold¶

In Backstage → Create → pick a service template (e.g. Node.js service, tagged blessed). In the scaffolder, the service values file path will default to services/<name>/helm-values-aws.yaml (legacy path). To place it under the team path, use the target path field and set: teams/payments/services/<service-name>.

Step 5 — Store team secrets¶

Write secrets to AWS Secrets Manager under the team path:

aws secretsmanager create-secret \
  --name "payments/database-url" \
  --secret-string '{"DB_URL":"postgres://..."}'

Reference from a Kubernetes ExternalSecret:

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: database-url
  namespace: team-payments
spec:
  secretStoreRef:
    name: team-payments-secrets    # the namespace-scoped SecretStore
    kind: SecretStore
  target:
    name: database-url
  data:
    - secretKey: DB_URL
      remoteRef:
        key: payments/database-url
        property: DB_URL

The team SecretStore can only read /payments/* paths — cross-team access is blocked at the IAM level.

Step 6 — Crossplane resources¶

When a team creates a Crossplane Claim in team-payments, Kyverno automatically: - Mutates: injects spec.parameters.team: payments (derived from namespace name) - Validates: blocks claims with no owner or costCenter

This ensures every AWS resource created by the team is tagged idp:team=payments for OpenCost showback. No manual tagging needed.

Step 7 — Grafana team folder¶

After the PR merges, the Grafana sidecar picks up the grafana-folder-team-payments ConfigMap in the monitoring namespace and creates a "Team — payments" folder.

Team members can: 1. Save dashboards to their folder via the Grafana UI 2. Create ConfigMaps labeled grafana_folder: "Team — payments" in monitoring to provision dashboards as code

DORA metrics per team¶

Tag all team service repos on GitHub with topic team:payments. The DORA exporter detects this and emits dora_deploy_frequency_per_day{service="orders-api",team="payments"}.

Alternatively, set TEAM_MAP in AWS Secrets Manager:

{"orders-api": "payments", "notifications-worker": "payments"}

Offboarding a team¶

# 1. Delete the team's services from teams/<name>/services/ (ArgoCD will prune)
# 2. Delete the namespace (cascades to all resources)
kubectl delete namespace team-payments

# 3. Remove the Terraform IAM role
terraform apply -var='team_eso_roles=[]'

# 4. Archive the scaffold PR branch or delete kubernetes/teams/payments/
# 5. Remove/archive the "payments" GitHub team in the org — the Backstage
#    Group entity is synced live and disappears on the next catalog refresh
#    once the GitHub team is gone (nothing to clean up in this repo)

Troubleshooting¶

Symptom	Likely cause	Fix
ArgoCD Application `payments-orders-api-dev` not created	Values file not at `teams/payments/services/orders-api/helm-values-aws.yaml`	Check path convention
`ExternalSecret` in error: `SecretStore not found`	IAM role ARN not set; SecretStore not created	Run `terraform apply` with `team_eso_roles`, re-run scaffold
Crossplane claim rejected: `owner is required`	Kyverno validate policy blocking	Add `spec.parameters.owner` to the claim
Grafana folder not appearing	Grafana sidecar not picking up ConfigMap	Check `kubectl get cm -n monitoring -l grafana_folder`
`team=unknown` on DORA metrics	Repo not tagged with `team:<name>` topic	Tag the repo or add to `TEAM_MAP`