Crossplane¶

Crossplane provisions per-service AWS resources (RDS, S3, Kafka topics, DynamoDB tables, SQS queues) as Kubernetes Claims, reconciled in-cluster by ArgoCD. It coexists with Terraform — see crossplane-vs-terraform.md for the boundary.

Why both¶

Terraform stays the right tool for the foundation: VPC, EKS cluster, IAM/OIDC, ECR registries, anything that must exist before Kubernetes is running. One-shot, platform-team-owned, applied from outside the cluster.

Crossplane is the right tool for day-2, app-team-requested infra. Claims live in Git alongside the service that consumes them, ArgoCD syncs on merge, Crossplane reconciles continuously, and drift is corrected without a human running terraform apply. No manual step between "PR merged" and "resource exists".

End-to-end example: provisioning an S3 bucket¶

Flow: Backstage scaffolder opens a PR to GitHub, ArgoCD syncs it and applies an S3Bucket Claim, Crossplane with provider-aws-s3 reconciles it, and IRSA creates the real S3 bucket in AWS

Engineer opens the Backstage portal, picks the S3 Object Bucket (Crossplane) template, fills in bucketName, ownerService, region, versioning, cost center.
Backstage opens a PR adding services/<ownerService>/claims/<bucketName>.yaml (an S3Bucket Claim) and a catalog-info-<bucketName>.yaml describing the resource as a Backstage Resource entity (dependencyOf the owning Component). It is not picked up automatically: after merging, register it via Catalog → Register existing component using the URL in the PR description.
Reviewer merges. The idp-services ApplicationSet in aws/argocd/app-of-apps.yaml already watches services/*/, so ArgoCD syncs the Claim within ~60 s.
The XRD xs3buckets.idp.platform matches the Claim; the Composition xs3bucket.aws creates a Bucket, BucketServerSideEncryptionConfig, BucketPublicAccessBlock, and BucketVersioning — all tagged idp:provisioner=crossplane, idp:owner=<team>, idp:cost-center=….
provider-aws-s3 assumes the IRSA role (terraform/iam-crossplane.tf, least-privilege inline policy scoped to arn:aws:s3:::idp-*) and creates the bucket in AWS.
Verify: kubectl get s3bucket <bucketName> -n services-dev → READY=True within ~60 s of merge.

Resources covered¶

Per-service resources (team self-service)¶

Resource	XRD kind	Claim kind	Template	`idp:team` tag
S3 bucket	`XS3Bucket`	`S3Bucket`	`s3-bucket-crossplane`	AWS resource tag
RDS Postgres	`XRDSInstance`	`RDSInstance`	`rds-database-crossplane`	AWS resource tag
MSK topic	`XKafkaTopic`	`KafkaTopic`	`kafka-topic-crossplane`	K8s label (MSK topics don't support AWS tags)
DynamoDB table	`XDynamoTable`	`DynamoTable`	`dynamodb-table-crossplane`	AWS resource tag
SQS queue	`XSQSQueue`	`SQSQueue`	`sqs-queue-crossplane`	AWS resource tag

All five Compositions live in aws/crossplane/compositions/. Adding a new resource type is a matter of dropping in another xrd.yaml + composition.yaml pair plus a matching scaffolder template.

Team label injection (Kyverno)¶

A pair of Kyverno policies in kubernetes/policies/kyverno/crossplane-team-label-policy.yaml enforce cost-attribution discipline on all Crossplane claims:

Mutate — when a Claim is created in a team-* namespace, Kyverno automatically adds spec.parameters.team: <teamName> (derived by stripping the team- prefix from the namespace). Teams never need to set this field manually.

Validate (Enforce) — rejects any Claim (in any namespace) that is missing spec.parameters.owner or spec.parameters.costCenter. This prevents untagged AWS resources that OpenCost cannot attribute.

# Verify the policies are active
kubectl get clusterpolicy crossplane-inject-team-tag crossplane-require-cost-tags

# See team tag on a provisioned S3 bucket
aws s3api get-bucket-tagging --bucket my-bucket | jq '.TagSet[] | select(.Key == "idp:team")'

Kyverno is installed by bootstrap-local.sh (Step 9b) and bootstrap.sh (Phase 3.8). If you see Kyverno admission errors, verify the kyverno-admission-controller deployment is Available in the kyverno namespace.

IAM: least-privilege provider roles¶

The Crossplane IRSA role (terraform/iam-crossplane.tf) uses one inline policy per resource family, each scoped to the minimum actions the Composition actually performs and restricted to idp-* ARN prefixes:

Policy	Key actions	ARN scope
`crossplane_s3`	CreateBucket, PutEncryptionConfiguration, PutBucketVersioning, …	`arn:aws:s3:::idp-*`
`crossplane_rds`	CreateDBInstance, DeleteDBInstance, CreateDBSubnetGroup, …	`arn:aws:rds:::db:idp-*`
`crossplane_kafka`	CreateTopic, DeleteTopic, DescribeCluster, …	`arn:aws:kafka:::cluster/idp-/`
`crossplane_dynamodb`	CreateTable, DeleteTable, UpdateContinuousBackups, …	`arn:aws:dynamodb:::table/idp-*`
`crossplane_sqs`	CreateQueue, DeleteQueue, SetQueueAttributes, …	`arn:aws:sqs:::idp-*`
`crossplane_tagging`	tag:TagResources, tag:GetResources, …	`*` (required by tagging API)

Why idp-* scope? XRD pattern fields enforce the same prefix on resource names, so a Claim can never request a name that falls outside the policy's resource scope.

Safety defaults in all Compositions¶

Behaviour	Value	Rationale
`deletionPolicy`	`Orphan`	Deleting a Claim does not delete the AWS resource. Prevents data loss from accidental `kubectl delete`. Manual decommission via `cleanup.sh` or AWS console required.
`skipFinalSnapshot` (RDS)	`false`	A final DB snapshot is taken before the instance is deleted.
`backupRetentionDays` (RDS)	`30` (configurable, 1–35)	Automated RDS backups retained 30 days by default.
`storageEncrypted` (RDS)	`true`	Always encrypted at rest.
`publiclyAccessible` (RDS)	`false`	VPC-only; never exposed to the internet.
`sqsManagedSseEnabled` (SQS)	`true`	Server-side encryption always on.
`pointInTimeRecovery` (DynamoDB)	`true`	PITR enabled by default.

Scaffolder templates¶

All five Crossplane templates follow the same flow:

Fill the form → Backstage generates a PR with two files:
- services/<ownerService>/claims/<name>.yaml — the Crossplane Claim
- services/<ownerService>/claims/catalog-info-<name>.yaml — Backstage Resource entity
Merge → ArgoCD syncs → Crossplane provisions the resource
Register the catalog-info-<name>.yaml via Backstage → Catalog → Register existing component (the exact URL is in the PR description and the task output). Until you do, the resource is not in the catalog.

DynamoDB composite keys: The template exposes optional rangeKey and rangeKeyType fields. Leave them blank for hash-only tables. When set, the Composition patches both attribute[0] (hash key) and attribute[1] (range key) definitions — both are required by the DynamoDB API.

Kafka clusterArn: The template validates the ARN format (^arn:aws:kafka:) and shows an in-form help note. To find your MSK cluster ARN:

aws kafka list-clusters --query "ClusterInfoList[*].ClusterArn" --output text

Bootstrap¶

# Phase 1 — Terraform provisions the IRSA role (validates ARN format before applying)
cd terraform && terraform apply
terraform output crossplane_aws_role_arn   # confirm ARN looks valid

# Phase 2 — bootstrap.sh substitutes the ARN and registers the ArgoCD stack
# (also validates the ARN starts with arn:aws:iam:: before substitution)
./scripts/bootstrap.sh

The deployment-runtime-config.yaml IRSA substitution is validated at bootstrap time — if terraform output returns a non-ARN value, bootstrap fails with a clear error rather than silently starting providers with a broken annotation.

Verifying¶

# All five providers Healthy
kubectl get providers.pkg.crossplane.io

# XRDs established (5 total)
kubectl get xrds

# ProviderConfig default present
kubectl get providerconfigs.aws.upbound.io

# Provider Pod has the IRSA role annotation
kubectl get pods -n crossplane-system -l pkg.crossplane.io/provider \
  -o jsonpath='{.items[*].spec.serviceAccountName}'

Drift correction¶

Crossplane reconciles continuously. To prove it:

# Delete the bucket directly in AWS (or via kubectl)
aws s3 rb s3://<bucketName> --force
# Within ~30s, Crossplane recreates it (because deletionPolicy: Orphan
# means Crossplane still owns the desired state even if the Claim persists)
kubectl get s3bucket <bucketName> -n services-dev -w

Decommissioning a resource¶

Because deletionPolicy: Orphan is set, deleting the Claim does not delete the AWS resource. To fully decommission:

# 1. Delete the Claim (removes Crossplane tracking only)
kubectl delete s3bucket <bucketName> -n services-dev

# 2. Manually delete the AWS resource
aws s3 rb s3://<bucketName> --force        # S3
aws rds delete-db-instance --db-instance-identifier <id> --skip-final-snapshot  # RDS
aws dynamodb delete-table --table-name <name>   # DynamoDB
aws sqs delete-queue --queue-url <url>           # SQS

# 3. Remove the claim YAML from the repo to prevent ArgoCD re-creating it
git rm services/<ownerService>/claims/<name>.yaml
git rm services/<ownerService>/claims/catalog-info-<name>.yaml
git commit -m "chore: decommission <name>"

Environment teardown uses cleanup.sh which automates steps 2 for all idp:provisioner=crossplane-tagged resources. See the Deployment Guide for details.

Local development¶

Crossplane runs in EKS only. The local Kind path stays unchanged — Docker- Compose-backed Postgres/Kafka are the substitutes for local dev. Cloud-only resources (real S3, DynamoDB, SQS) are not testable locally; write integration tests that hit AWS or LocalStack in CI instead.