How to deploy 100 microservices with one Helm chart

At tens of services, CI/CD rarely fails loudly—it degrades. This case study maps the delivery contract that kept one Universal Helm Chart workable toward a 60–90+ service fleet. Title “100” is a planning horizon, not a lab measurement.

The contract (one job per layer)

Scaling failed when every layer did a little of everything. The working model assigns one job:

• Infra repo (Terragrunt/Terraform or equivalent — Illustrative control-plane shape): cloud, cluster, Argo CD lifecycle
• Service repo: build, test, publish image; include one shared GitLab CI template
• Bridge GitOps repo: environment-specific Helm values and wrapper Chart.yaml pins
• Universal Helm chart: Deployment/Service/Job/HPA/HTTPRoute contract as a platform API
• Argo CD ApplicationSets: discover apps by path and reconcile

App teams ship code. Platform evolves the contract. Argo does not author values.

Repository trees

Bridge GitOps (Verified shape):

gitops-bridge/
├── .ci/.gitlab-ci.yml          # shared template included by app repos
├── argocd/
│   ├── appset-dev.yaml         # path: environments/dev/*
│   ├── appset-prod.yaml
│   └── appproject-prod.yaml
├── environments/
│   ├── dev/<app>/{Chart.yaml,values.yaml}
│   ├── prod/<app>/{Chart.yaml,values.yaml}
│   └── infra/<component>/...   # Gateways, Vault, runners — not product apps
├── templates/app-onboard/
└── docs/

Service repo (Verified pattern): application code + Dockerfile + a thin .gitlab-ci.yml that only includes the shared template (pin ref to a tag/SHA in production).

Do not count infra/ paths as microservices. Battle inventory is smaller than case-study fleet claims; see examples/deploy-100-microservices/METHODOLOGY.md.

Service onboarding

1. Create the service repository and include the shared CI template.
2. Add environments/dev/<app>/ (and prod/ when ready) with a wrapper that depends on application.
3. Push an image. ApplicationSet creates dev-<app> / prod-<app> from path discovery.

Case-study claim: onboarding dropped from roughly a day to under thirty minutes after the scaffold + shared template existed. Treat that as an operator report until you run the stopwatch methodology in the evidence pack.

Per-environment values and image metadata

Wrappers nest runtime config under application:. CI may mutate only application.image and application.imageTag (battle finding: the first CI job wrote the wrong values level and broke rollouts).

# environments/prod/demo-api/Chart.yaml
dependencies:
  - name: application
    version: "0.4.3"
    repository: https://chaser100.github.io/u-helm-chart

# environments/prod/demo-api/values.yaml (abbreviated — chart 0.4.3 keys)
application:
  image: ghcr.io/example-org/demo-api
  imageTag: "1.0.0"
  route:
    enabled: true
    gateway: external
    gatewayNamespace: kgateway-system
    sectionName: https-wildcard
    hostname: demo-api.example.com
    path: /
    pathMatchType: PathPrefix
    backendWeight: 100
    timeouts:
      request: 60s
      backendRequest: 55s

Full wrappers: examples/deploy-100-microservices/bridge-layout/. Simple mode maps route.hostname → HTTPRoute spec.hostnames[0]. For multi-host / multi-rule routes on 0.4.3, set route.spec (complete HTTPRoute.spec) instead of inventing top-level route.hostnames / route.rules shorthand keys.

Promotion

• dev: push to the service dev branch → auto GitOps write into environments/dev/<app>
• prod: merge to main → manual gitops_update for the git_commit path

Copy route semantics carefully; do not blindly copy digests between environments. Chart version bumps are platform MRs rolled by canary rings.

Chart version rollout

1. Platform ships a new application chart version.
2. Bump the wrapper dependency on a canary app/env first.
3. Confirm Deployment rollout and HTTPRoute Accepted=True / ResolvedRefs=True.
4. Roll the pin across ApplicationSet paths; inventory remaining pins so a partial upgrade does not leave a mixed fleet.

Rollback under selfHeal: true: cluster mutations (including kubectl rollout undo) are overwritten by the Git desired state. Fix the bridge first—revert the wrapper Chart.yaml pin or re-pin imageTag—then verify Application Synced/Healthy, HTTPRoute Accepted/ResolvedRefs, and health HTTP 200. Live prod rollback drills were deferred in battle results (residual risk).

Concurrency-safe GitOps writes

At fleet scale, non-fast-forward pushes are expected. The battle template retries five times with git fetch + git rebase and no sleep/jitter (Verified baseline). For production hardening, use bounded full-jitter sleep and fail closed (Illustrative):

# CI GitOps updater: bounded retry with full-jitter sleep
MAX_ATTEMPTS=5 BASE_SLEEP_SEC=2 CAP_SLEEP_SEC=30

After max attempts the pipeline fails; the next green run retries from a fresh clone. Rebase conflicts on the same values path also fail closed (no auto-resolve)—serialize hot paths with GitLab resource_group / CODEOWNERS. Adopt jitter (or serialization) before high parallel gitops_update load; battle had neither sleep nor jitter.

ApplicationSets

Path generators create one Application per discovered directory. Sanitized excerpt:

generators:
  - git:
      repoURL: https://gitlab.example.com/platform/gitops-bridge.git
      revision: HEAD
      directories:
        - path: environments/dev/*
template:
  metadata:
    name: 'dev-{{ .path.basenameNormalized }}'
  spec:
    syncPolicy:
      automated: { prune: true, selfHeal: true }

Put CRDs, SecretStores, and GatewayClass in an infra ApplicationSet (or lower sync waves) before application routes. Prefer pinning prod targetRevision to a Git SHA/tag for freeze windows (Illustrative vs battle HEAD). With prune: true, deleting an environments/<env>/<app> path removes the Application and its cluster objects—gate path deletes with MR review / CODEOWNERS.

kgateway at fleet scale

• Centrally managed Gateways: platform owns listeners, TLS Secret refs, and sectionName; apps set route.gateway / route.gatewayNamespace only
• Generated HTTPRoutes: published chart 0.4.3 route.* shorthand (hostname/path/backendWeight/timeouts) renders per-app routes from bridge values
• Hostname/path collisions: registry + onboard MR checks; see gateway/HOSTNAME-REGISTRY.md
• ReferenceGrant: platform-owned; apps never invent cross-namespace grants
• Policy attachment: kgateway-specific CORS/rate-limit via labeled extraManifests — chart 0.4.3 has no portable route.cors / route.rateLimit
• Progressive delivery: simple mode uses route.backendWeight (single Service). Multi-backend canary weights: set route.spec.rules[].backendRefs[] with weights (chart 0.4.3), or a labeled extraManifests HTTPRoute; verify parent Gateway Accepted before shifting traffic
• Status verification: Application Synced/Healthy + HTTPRoute Accepted/ResolvedRefs + health HTTP 200
• ingress-nginx migration waves: Gateway live → shadow routes → pilot cutover → ownership-group waves → retire Ingress last

Hands-on conversion details: Ingress NGINX to kgateway and kgateway migration patterns.

Ownership, exceptions, and when one chart stops

Rare exceptions stay labeled (networking.contract/kind: kgateway-specific). If escape hatches dominate, or multi-workload values become operationally unreadable, use a dedicated chart. Decision boundaries: examples/deploy-100-microservices/WHEN-ONE-CHART-STOPS.md and One Helm chart vs many application charts.

Failure modes and operational cost

• Concurrent bridge writers / missing jitter (bounded retry + full-jitter or resource_group)
• Rebase conflicts on the same values path (fail-closed; human merge)
• Accidental path delete under prune: true (MR/CODEOWNERS gate)
• Wide blast radius from a bad shared chart release (canary rings + pin inventory)
• Hostname/path collisions on shared Gateways
• Escape-hatch debt and mixed chart pins after partial rollouts
• Image Updater writing both environments (separate prod CR / hybrid git_commit)

Full table: examples/deploy-100-microservices/FAILURE-MODES.md. Residual: live prod rollback drill deferred; battle concurrency has no jitter—harden before fleet-parallel writes.

What “100” means here

The title is a planning horizon. The delivery model is the same at six battle apps or an operator-reported 60–90+ fleet: one chart contract, one CI include, one bridge surface, ApplicationSet discovery, and honest labels on every quantitative claim. Until you run the methodology in METHODOLOGY.md, publish outcomes as case-study reports—not as lab SLOs.

Official documentation

• Helm Charts / helm dependency
• Argo CD ApplicationSet
• Kubernetes Gateway API
• Universal Helm Chart / Artifact Hub

One chart vs many → · GitOps repository structure → · Why one Helm chart is enough →

Related in this series

GitOps repository structure → · One chart vs many → · Why one Helm chart is enough → · Internal Developer Platform → · kgateway migration patterns → · Ingress NGINX to kgateway →