Golden Path: From Template to Running Service¶

The golden path is the single, opinionated workflow every developer follows to ship a service. No decisions about CI, container registries, deployment targets, or monitoring — the platform handles those.

The Journey (end-to-end)¶

Local (Kind)¶

Developer                Platform
─────────                ────────
1. Open Backstage (http://backstage.idp.local)
2. Choose template ──────> Scaffolder creates GitHub repo + README
3. Clone & code          > CI workflow is pre-wired (test + smoke-check)
4. git push ─────────────> GitHub Actions: install → test → docker build → /healthz check
5. Push image to registry
   docker push localhost:5003/<name>:latest
6. Open Backstage ───────> Create → "Deploy Service to local Kind cluster"
                         > idp:deploy-local action runs helm upgrade --install
7. Done — service live   > http://<name>.idp.local

AWS (planned)¶

Developer                Platform
─────────                ────────
1. Open Backstage
2. Choose template ──────> Scaffolder creates GitHub repo + README
3. Clone & code          > CI/CD workflow is pre-wired
4. Add AWS secrets ──────> AWS_ROLE_ARN in repo settings
5. git push ─────────────> GitHub Actions triggers
                         > Build Docker image
                         > Push to Amazon ECR
                         > helm upgrade --install on EKS
                         > Deployment verified
6. Done — service live   > Logs/metrics in CloudWatch + Grafana

Step-by-Step for Developers¶

1. Pick a template in Backstage¶

Open the Backstage portal and click Create → select one of:

Service templates

Template	Language	Default Port
Node.js Service	Express	3000
Python FastAPI Service	FastAPI + uvicorn	8000
Go Service	Go	8080
Ruby Sinatra Service	Ruby / Sinatra	4567
JVM (Java/Spring Boot) Service	Java 21 / Spring Boot / Gradle	8080
LLM App (Python + Langfuse)	FastAPI + Claude, tracing pre-wired	8000

Fill in name, description, owner, and GitHub repo. Click Create.

The repository must not exist yet. If the name is already taken, the form says so when you click Next on the repository step ("acme/payments already exists on GitHub. Choose another repository name"), before anything runs. Previously that only surfaced as a failed Publish to GitHub step partway through the run. If the check itself can't reach GitHub it lets you continue, and the publish step reports a real conflict as before.

Backstage will: - Fetch the skeleton and render it with your values - Publish the repo to GitHub, tagged with the idp-service topic (idp-app for mobile apps) so catalog discovery keeps finding it - Register the component and API in the catalog

How a scaffolded repo stays in the catalog¶

Registration is not a one-off write. catalog.providers.github.idpOrg (in app-config.yaml, with an AWS counterpart in app-config.aws.yaml) polls the GitHub org every 15 minutes and ingests the catalog-info.yaml from the main branch of every repo carrying the idp or idp-app topic. Scaffolded repos therefore appear on their own, and edits to a repo's catalog-info.yaml flow back into Backstage without touching this repo.

Two consequences worth knowing:

Remove the topic and the service leaves the catalog — and, because the DORA exporter cross-checks the catalog, it drops off the DORA dashboards too (see DORA & FinOps § Which services appear).
A repo that isn't discovered can still be registered by hand via a catalog.locations URL entry, which is how this repo's own hand-written services are listed.

Per-service infra (Crossplane, AWS)

Template	Resource	Notes
S3 Object Bucket (Crossplane)	S3 bucket with encryption + public-access block	Versioning configurable
RDS Postgres (Crossplane)	RDS instance + connection Secret	30-day backup retention default; `backupRetentionDays` param
Kafka Topic (Crossplane)	MSK topic on an existing cluster	Requires MSK cluster ARN; form validates `arn:aws:kafka:` prefix
DynamoDB Table (Crossplane)	DynamoDB table with PITR	Optional `rangeKey` + `rangeKeyType` for composite primary keys
SQS Queue (Crossplane)	SQS queue (FIFO opt-in)	SSE always enabled

Each template opens a PR with two files: - services/<ownerService>/claims/<name>.yaml — the Crossplane Claim - services/<ownerService>/claims/catalog-info-<name>.yaml — the Backstage Resource entity for it

ArgoCD syncs on merge, Crossplane provisions the AWS resource — no terraform apply step. The Resource entity is not auto-discovered: after merging, register it via Catalog → Register existing component with the URL from the PR description. See crossplane.md for the full flow.

The legacy Terraform-PR templates (s3-bucket, rds-database, kafka-topic) still exist for callers mid-migration. Pick the Crossplane variant for new resources. They write into their own directories (e.g. terraform/infra/s3-buckets/<name>/) rather than the repo root, and need the same manual catalog registration after merge.

2. Clone your new repo¶

git clone https://github.com/moatazeldebsy/<service-name>
cd <service-name>

3. Implement your service¶

The skeleton contains:

<service-name>/
├── src/                     # Application code + tests
├── Dockerfile               # node:22-alpine, python:3.13-slim, or Go on distroless static
├── README.md                # Auto-generated with endpoints, commands, deploy steps
├── openapi.yaml             # API spec (served at /openapi.json)
├── helm-values-local.yaml   # Kind / nginx overrides
├── catalog-info.yaml        # Backstage component registration
├── api-info.yaml            # Backstage API registration
├── sonar-project.properties # SonarCloud config (coverage report paths)
├── mkdocs.yml + docs/       # TechDocs
└── .github/
    └── workflows/
        ├── ci.yml              # lint + vuln scan, test + 70% coverage gate, container smoke test, publish
        └── contract-check.yml  # OpenAPI contract validation on PRs

The skeleton carries only the local values file. The AWS values (helm-values-aws.yaml) come from the template's platform-values/ directory and land in the GitOps pull request the scaffolder opens, not in your service repo's initial commit. There is no bare helm-values.yaml anywhere — no ApplicationSet reads one.

Note that helm-values-aws.yaml ships with ingress.enabled: false. Scaffolded services get no ALB by default; opt in explicitly if the service needs public ingress.

Optional parameters have real defaults. The scaffolder's "Infrastructure" parameter page (port, cost centre) can be skipped — templates apply port | default(8080) and costCenter | default("eng-platform") at the step level. Skipping it used to emit a bare EXPOSE with no port and break the new repository's first CI run; that is fixed, so you only need that page when the service genuinely differs from the defaults.

Conventions (enforced by the platform):

Convention	Value
Liveness path	`GET /healthz` → `200 {"status":"ok"}`
Readiness path	`GET /ready` → `200 {"status":"ready"}`
Metrics path	`GET /metrics` → Prometheus text format
Logs	Structured JSON to stdout
Namespace	`services`

4. Push to trigger CI¶

git add .
git commit -m "feat: initial implementation"
git push origin main

GitHub Actions (test job on ubuntu-latest) will: 1. Install dependencies (npm install / pip install) 2. Run tests (passes if no tests exist yet) 3. Build the Docker image (smoke check) 4. Start the container and curl /healthz and /ready

5. Deploy to local Kind¶

# Build and push the image to the local registry
docker build -t localhost:5003/<name>:latest .
docker push localhost:5003/<name>:latest

Then in Backstage → Create → Deploy Service to local Kind cluster: - Pick the service from the catalog - Set image tag (latest) - Click Create

Or via CLI:

helm upgrade --install <name> ./helm/service-template \
  --namespace services --create-namespace \
  --set image.repository=localhost:5003/<name> \
  --set image.tag=latest \
  --values services/<name>/helm-values-local.yaml

Access the service at http://<name>.idp.local (add to /etc/hosts if needed).

6. Deploy to AWS (when ready)¶

Scaffolding already opened a GitOps pull request against the platform repo adding services/<name>/helm-values-aws.yaml, whose image repository is the ECR repository the scaffolder created. Merge it and the idp-services ApplicationSet deploys the service to services-dev. The service's ci.yml pushes every main build to that ECR repository (and to GHCR) using the push-only role the scaffolder created for it; with a GH_PAT repository secret it also bumps the image tag in the platform repo's values file. See Adding AWS CD to a Scaffolded Service.

Crossplane claims and secrets requested later (the *-crossplane, add-secret and rds-database templates) land in services/<name>/claims/ and services/<name>/secrets/ of the platform repo and are applied by the idp-service-resources ApplicationSet (AWS only).

7. Monitor your service¶

Local: Grafana → http://grafana.idp.local (admin/admin)
AWS: CloudWatch → Log Groups → /aws/containerinsights/idp-mvp/application
Metrics: Grafana → IDP Services dashboard

Progressive delivery¶

Every service deploys through the same chart, which renders either a plain Deployment or an Argo Rollout. Opt in with the Enable Progressive Delivery scaffolder template — it opens a PR adding services/<name>/helm-values-rollout-<env>.yaml, which the ArgoCD ApplicationSets already list as an optional second values file. Nothing has to be hand-merged.

Which strategy¶

	Canary	Blue-green
Traffic	Shifts gradually (20% → 50% → 100% by default)	All at once, on promotion
Extra capacity	One canary ReplicaSet, sized to the step weight	A full second stack
Analysis	Runs at each step; failure aborts and rolls back	Runs before promotion
Rollback	Abort mid-rollout; blast radius limited to the canary weight	Instant — active Service points back at the old ReplicaSet
Good for	Stateless HTTP services with useful per-request metrics	Session-affine or stateful services, expensive analysis, anything where you want to verify the new version end to end before any user sees it

Blue-green doubles the pod count during promotion. On a single-node local cluster set rollout.blueGreen.previewReplicaCount: 1, or the preview stack will evict platform components.

Blue-green also keeps a human in the loop by default (autoPromotionEnabled: false): the new version comes up fully and is reachable on the -preview Service, and you promote with

kubectl argo rollouts promote <service> -n <namespace>

Rollback thresholds¶

rollout.analysis.errorRateThreshold and latencyThresholdSeconds are passed to the cluster-scoped http-error-rate AnalysisTemplate as arguments, so the thresholds are per-service while the PromQL stays in one place. They are fractions, not percentages — 0.01 is 1%. The scaffolder form takes a percent and converts, so this only matters when hand-editing.

The chart ships a values.schema.json covering the rollout block, so a typo like strategy: blue-green fails at helm lint rather than producing a Rollout with no strategy — which Argo accepts and then never progresses.

Conventions Reference¶

Convention	Local	AWS
Namespace (legacy services)	`services`	`services`
Namespace (team services)	`team-<name>`	`team-<name>`
Service values path (legacy)	`services/<name>/helm-values-local.yaml`	`services/<name>/helm-values-aws.yaml`
Service values path (team)	`teams/<name>/services/<svc>/helm-values-local.yaml`	`teams/<name>/services/<svc>/helm-values-aws.yaml`
Image registry	`localhost:5003/<name>`	`<account>.dkr.ecr.<region>.amazonaws.com/idp-mvp/<name>`
Image tag	`latest` (local push)	`<git-sha-short>`
Ingress class	`nginx`	`alb` (but `ingress.enabled: false` by default — scaffolded services get no ALB unless opted in)
Replicas	1	2
CPU request	50m	100m
Memory request	32Mi	128Mi
Team label (K8s)	`team: <name>`	`team: <name>`
Team tag (AWS)	`idp:team=<name>` (Kyverno auto-injects on Crossplane claims)	same
Secret path	n/a (local uses ConfigMaps)	`/<teamName>/<secret-name>` in Secrets Manager
Repo field in a template that creates a repo	`ui:field: NewRepoUrlPicker` (the stock picker plus a "name must not exist" check via `/api/idp-repo-check`)	same
Repo field in a template that opens a PR on an existing repo	`ui:field: RepoUrlPicker`. Never `NewRepoUrlPicker`, which would reject every existing repo	same

CLI Alternative (without Backstage)¶

The CLI scaffolder covers the three base runtimes — nodejs, python, go. The specialist templates (Ruby, JVM, React, LLM App, MCP server, KAgent agent) are Backstage-only by design: they carry skeletons the CLI's local generator does not duplicate, which keeps the two implementations from drifting.

# Scaffold — auto-detects Backstage; falls back to local generation
idp scaffold service --name my-service --type nodejs

# Force local generation (offline / pre-Backstage)
idp scaffold service --name my-service --type nodejs --local

# Preview what would be generated without writing anything
idp scaffold service --name my-service --type nodejs --dry-run

# Deploy locally
docker build -t localhost:5003/my-service:latest services/my-service/
docker push localhost:5003/my-service:latest
helm upgrade --install my-service ./helm/service-template \
  --namespace services --create-namespace \
  --set image.repository=localhost:5003/my-service \
  --set image.tag=latest \
  --values services/my-service/helm-values-local.yaml

CLI: Scaffold a test suite¶

The idp scaffold test-suite command generates a ready-to-run QA suite alongside your service. It supports 16 test types covering every pyramid layer.

# Playwright E2E suite
idp scaffold test-suite --name my-svc-e2e --type playwright --service my-service

# k6 load test — 20 VUs, 2 min, p95 < 400 ms
idp scaffold test-suite --name my-svc-load --type k6 --service my-service \
  --vus 20 --duration 2m --p95 400

# OWASP ZAP DAST security scan
idp scaffold test-suite --name my-svc-sec --type zap --service my-service \
  --scan-type baseline

# Preview before writing
idp scaffold test-suite --name my-svc-e2e --type playwright --service my-service --dry-run

Generated suites land in test-suites/<name>/ and are auto-registered in the Backstage catalog via catalog-info.yaml.

See CLI Reference for all 16 test suite types and their flags.