Building an Internal Developer Platform with Helm

Treat the platform as a product with a stable contract: one reusable Helm chart, a bridge GitOps repo for desired state, and shared Gateway API infrastructure. Developers ship services without editing Deployment YAML.

Platform product, tooling, and process

Keep three layers distinct. When they blur, the platform becomes either a ticket desk or an unmanaged toolbox.

Product is the paved road: create service, build image, promote values, sync, observe. Measure it with time-to-first-deploy, percent of services on the golden path, and support ticket rate.

Tooling is the Helm chart, Argo CD ApplicationSets, shared Gateway and TLS, External Secrets, and policy engines. Tools change; the application contract should stay stable.

Process is RACI, the exception path, chart upgrade rings, and scorecard reviews. Process without tooling does not scale. Tooling without process becomes tribal knowledge.

Users and jobs-to-be-done

• Application developer: ship a service with image, probes, env, and hostname without editing Deployment YAML.
• Service owner: know who owns runtime health, on-call, and scorecard debt.
• Platform engineer: evolve the chart contract, Gateways, and golden-path defaults without per-repo forks.
• SRE: consume SLIs (sync latency, route Accepted, error rate) and define budgets outside this article.
• Security / compliance: verify non-root, secret references, NetworkPolicy posture, and labeled escape hatches.

Architecture (repo-native)

┌──────────────┐   catalog request    ┌─────────────────────┐
│ Developer    │ ───────────────────► │ Service catalog /   │
│ (self-serve) │                      │ scaffolding job     │
└──────────────┘                      └─────────┬───────────┘
                                                │ generates
          ┌─────────────────────────────────────┼──────────────────────────┐
          ▼                                     ▼                          ▼
 ┌─────────────────┐                 ┌──────────────────┐      ┌────────────────────┐
 │ Service repo    │  CI: build/push │ Bridge GitOps    │      │ Shared platform    │
 │ + shared CI     │ ──────────────► │ values + Chart   │      │ Gateway / TLS /    │
 │ template        │  imageTag only  │ pin (application)│      │ ESO / policies     │
 └─────────────────┘                 └────────┬─────────┘      └─────────┬──────────┘
                                              │ ApplicationSet            │
                                              ▼                           ▼
                                     ┌────────────────────────────────────────────┐
                                     │ Cluster: Deployment/Service/HTTPRoute via │
                                     │ Universal Helm Chart + kgateway listeners │
                                     └────────────────────────────────────────────┘

Ownership notes and the same diagram live in the evidence pack under examples/internal-developer-platform/.

Service catalog inputs

The catalog form is the public API of the platform. Minimum fields (see examples/internal-developer-platform/catalog/):

• serviceName, ownerTeam, language (fastapi|go|nodejs|…)
• environments (dev, prod), replicas profile
• route.hostname, route.gateway / gatewayNamespace / sectionName
• secretRefs[] (Secret name + keys — never literal values)
• sloTier (maps to resources/HPA defaults)

# catalog/template-inputs.example.yaml (abbreviated)
serviceName: payments-api
ownerTeam: payments
language: fastapi
environments: [dev, prod]
route:
  hostname: payments-api.dev.example.com
  gateway: external
  gatewayNamespace: kgateway-system
  sectionName: https-wildcard
secretRefs:
  - secretName: payments-api-secrets
    keys: [DATABASE_URL]

Golden path and reusable Helm contract

The paved road for a conventional HTTP service:

1. Scaffold the service repo with the shared CI template.
2. Create the bridge path environments/<env>/<service>/ with a wrapper Chart.yaml that pins application 0.4.3 and nested application: values.
3. Platform defaults inject securityContext, probes, resources, and route.* pointing at the shared Gateway.
4. CI updates only application.image / application.imageTag. Wrong values nesting breaks promotions.
5. Argo CD ApplicationSet discovers the path; sync produces Deployment, Service, and HTTPRoute.

# golden-path wrapper Chart.yaml
dependencies:
  - name: application
    version: "0.4.3"
    repository: https://chaser100.github.io/u-helm-chart

# values.yaml (nested)
application:
  replicaCount: 2
  image: ghcr.io/example-org/payments-api
  imageTag: "1.0.0"
  serviceAccount:
    create: true
    automount: false
  securityContext:
    runAsNonRoot: true
    runAsUser: 10001
    allowPrivilegeEscalation: false
    readOnlyRootFilesystem: true
    capabilities:
      drop: ["ALL"]
  route:
    enabled: true
    gateway: external
    gatewayNamespace: kgateway-system
    sectionName: https-wildcard
    hostname: payments-api.dev.example.com
  envSecrets:
    enableEnv: true
    envs:
      - name: DATABASE_URL
        secretName: payments-api-secrets
        secretKey: DATABASE_URL

Golden-path values and wrapper live under examples/internal-developer-platform/ (chart pin 0.4.3).

GitOps workflow

Four layers, one job each (from the battle delivery model):

• Infra repo owns the cluster and Argo CD control plane.
• Service repo builds, tests, and publishes the image.
• Bridge repo holds desired runtime state (image tag, route, resources).
• Argo CD reconciles and rolls out.

Bridge layout and ownership live in examples/gateway-api-helm-contract/BRIDGE-GITOPS-LAYOUT.md. The IDP delivery contract (ApplicationSet discovery, nested application.imageTag CI updates, and jittered GitOps push) is in examples/internal-developer-platform/DELIVERY-CONTRACT.md with matching excerpts under ci/ and gitops/. At scale, GitOps push conflicts need bounded fetch+rebase+jitter retries in the CI updater. Without that, demos look stable and production goes flaky.

kgateway / Gateway API as a platform service

Treat ingress as shared infrastructure, not an app-team CRD playground.

• Platform owns GatewayClass, shared Gateways, listeners (including allowedRoutes tenancy), TLS certificateRefs, DNS for platform wildcards, and ReferenceGrant where cross-namespace Service/Secret refs are required.
• App teams own self-service route.* inputs through bridge values only: hostname(s), path matches, and optional weights for canary.
• Apps attach to named listeners (sectionName) only when the Gateway listener allowedRoutes permits their namespace. Chart application 0.4.3 has no gatewayResource key, so shared Gateways stay platform-owned manifests (lab patterns live under examples/gateway-api-helm-contract/). Cross-namespace parentRefs are authorized by listener allowedRoutes, not by ReferenceGrant.
• kgateway-specific CRDs (TrafficPolicy, BackendConfigPolicy) go through labeled extraManifests with networking.contract/kind: kgateway-specific. Chart 0.4.3 has no route.cors or route.rateLimit keys. Do not invent them.
• Success means HTTPRoute Accepted=True and ResolvedRefs=True, plus HTTPS smoke on /health.
• For a controlled ingress-nginx migration, convert with ingress2gateway, review, flip Helm from ingress.* to route.*, and keep dual-stack rollback until cutover is proven. See the migration guide.

Policy, secrets, observability

• Policy: baseline securityContext in golden-path values. OPA/Kyverno may enforce non-root and ban NodePort/hostPath. Chart package defaults are empty, so enforcement belongs in platform defaults or admission.
• Secrets: only Secret references (envSecrets) or ExternalSecrets-managed Secrets. Never commit literals.
• Observability: optional chart serviceMonitor; platform scrapes; apps expose /metrics and health paths. Route Accepted and Argo sync are platform SLIs.

Operating model (RACI summary)

• Responsible — app team: image, allow-listed values overrides, on-call for app errors.
• Accountable — platform: chart version, Gateway/TLS, ApplicationSet, golden-path defaults.
• Consulted — SRE: SLI definitions, scorecard thresholds, migration risk.
• Informed — security: escape-hatch audits, secret posture.

The full matrix is in the evidence pack (examples/internal-developer-platform/).

Scorecards, SLOs, and adoption metrics

Measurable success (targets are platform-local; publish your own numbers):

• Time from catalog submit to first healthy dev deploy (battle path: under 30 minutes after scaffolding)
• Percent of services on pinned chart version N or N-1
• Percent of HTTPRoutes with Accepted/ResolvedRefs true
• Golden-path coverage vs exception tickets per month
• Mean chart upgrade ring duration (canary → fleet)

Watch for mixed chart fleets after partial upgrades, escape-hatch sprawl, CI updating the wrong values nesting, and apps creating private Gateways.

Exceptions, upgrades, feedback loops

1. Exception: file a request with blast radius, labeled extraManifests, and an expiry date. If most of the release is escape hatches, move to a dedicated chart.
2. Upgrade: platform ships a new chart version, canary the wrapper pin, verify Deployment + HTTPRoute, roll ApplicationSet paths, then inventory pins. Rollback means revert the wrapper dependency and sync.
3. Feedback: quarterly scorecard review, catalog NPS, and a single backlog for contract changes (not per-repo forks).

End-to-end self-service flow (generated artifacts)

1. Developer submits a catalog request (template-inputs.example.yaml).
2. Scaffold job creates the service repo, bridge Chart.yaml/values.yaml, and namespace Secret placeholder refs.
3. First push builds the image; CI writes imageTag into the bridge.
4. Argo syncs. The verify script/golden asserts Deployment, Service, HTTPRoute, and runAsNonRoot: true.
5. Developer receives dashboard links: Argo app, route hostname, runbook.

Artifact tree: examples/internal-developer-platform/golden-path/ plus golden/ from the verify script.

Build vs buy — when Helm alone is insufficient

Helm + GitOps is enough for conventional Deployment/Service/(Ingress|HTTPRoute) fleets with rare labeled exceptions.

Buy or build a portal when catalog UX, RBAC across many teams, and scorecards need a product surface (Backstage, Port, and similar). Keep Helm as the runtime contract underneath.

Dedicated charts or operators win when CRDs dominate the release, multi-workload values explode, or compliance freezes templates off the platform train. See the decision matrix.

Helm is not a secrets manager, a policy engine, or a progressive-delivery controller. Compose External Secrets, admission policy, and Argo Rollouts/Flagger beside the chart.

Validation evidence

• Battle test (2026-07-23): six languages × dev/prod = twelve Argo apps Synced/Healthy; HTTPRoutes Accepted/ResolvedRefs through external kgateway + wildcard TLS (BATTLE_TEST_RESULTS.md).
• Local pack: examples/internal-developer-platform/ — golden-path wrapper against chart 0.4.3 (helm lint / helm template).
• Related packs: examples/one-chart-vs-many/, examples/gateway-api-helm-contract/.

Official documentation

• Helm Chart Best Practices
• Argo CD — Helm
• Kubernetes Gateway API
• kgateway docs
• Universal Helm Chart / Artifact Hub

Why one Helm chart is enough → · One chart vs many → · Ingress NGINX to kgateway → · Gateway API examples → · Browse examples →

Related in this series

Platform engineering with Helm → · GitOps repository structure → · Deploy 100 microservices → · Kubernetes Secrets best practices → · One chart vs many → · Why one Helm chart is enough → · Gateway API examples → · Ingress NGINX to kgateway →