GitOps repository structure for Kubernetes applications

Separate application source, deployment values, environments, and platform configuration with a bridge GitOps repository, Argo CD path generators, and a pinned Universal Helm Chart dependency.

Four topologies

• Monorepo: source, charts, and env values in one remote. Fast for small fleets; noisy CODEOWNERS and wide merge blast radius as service count grows.
• Repository-per-environment: separate remotes for dev/prod. Hard prod ACLs; trees and chart pins drift unless promotion is automated.
• Repository-per-team: each team owns values (and often charts). Autonomy is high; platform contracts and ApplicationSets fragment.
• Bridge GitOps repository (Verified default): service repos build and push images; a bridge repo holds environments/{env}/{app} wrappers and Argo ApplicationSets; infra-as-code owns the control plane.

Full trade-off table: examples/gitops-repository-structure/TOPOLOGY-COMPARISON.md.

Layer contract

Service repo          Bridge GitOps repo              Cluster
─────────────         ──────────────────              ───────
app + Dockerfile  →   environments/<env>/<app>/
shared CI include →   Chart.yaml → application@0.4.3  → Argo ApplicationSet
build/push image  →   values.yaml (image + imageTag)  → Synced/Healthy
                      argocd/appset-*.yaml
                      environments/infra/...            → Gateway / SecretStores

Platform owns ApplicationSets, AppProjects, the shared CI template, and the universal chart contract. App teams own the service repository and MRs that change their values. Argo CD reconciles; it does not author desired state.

Concrete bridge tree

gitops-bridge/
├── .ci/.gitlab-ci.yml
├── argocd/
│   ├── appset-dev.yaml
│   ├── appset-stage.yaml          # Illustrative ring
│   ├── appset-prod.yaml
│   └── appproject-prod.yaml
├── environments/
│   ├── dev/demo-api/{Chart.yaml,values.yaml}
│   ├── stage/demo-api/{Chart.yaml,values.yaml}
│   ├── prod/demo-api/{Chart.yaml,values.yaml}
│   └── infra/shared-gateway/...
└── templates/app-onboard/

Minimal checked-in tree: examples/gitops-repository-structure/bridge-tree/. Battle-lab path counts (Verified): 13 under dev/, 6 under prod/, 19 under infra/. Do not equate those with case-study “60+/90+ services” claims—those remain operator-reported, not re-measured in this pack.

Wrapper Chart.yaml and values contract

# Full files:
# examples/gitops-repository-structure/bridge-tree/environments/prod/demo-api/{Chart.yaml,values.yaml}
dependencies:
  - name: application
    version: "0.4.3"
    repository: https://chaser100.github.io/u-helm-chart

# values.yaml (abbreviated — image/route/envSecrets only)
application:
  image: ghcr.io/example-org/demo-api
  imageTag: "1.2.0"
  route:
    enabled: true
    gateway: external
    gatewayNamespace: kgateway-system
    sectionName: https-wildcard
    hostname: demo-api.example.com
  envSecrets:
    enableEnv: true
    envs:
      - name: DEMO_API_DB_PASSWORD
        secretName: demo-api-runtime
        secretKey: db-password

CI may mutate only application.image and application.imageTag (Verified contract). Probes, resources, route hostnames, and secret references stay in Git as reviewed config. Runtime excerpt: examples/gitops-repository-structure/ci/values-runtime.excerpt.yaml.

Argo CD ApplicationSet (path generator)

# Abbreviated Verified shape — full file:
# examples/gitops-repository-structure/argocd/applicationset-dev.excerpt.yaml
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: appset-dev
  namespace: argocd
spec:
  generators:
    - git:
        repoURL: https://gitlab.example.com/platform/gitops-bridge.git
        revision: HEAD
        directories:
          - path: environments/dev/*
  template:
    metadata:
      name: 'dev-{{ .path.basenameNormalized }}'
    spec:
      project: dev
      source:
        repoURL: https://gitlab.example.com/platform/gitops-bridge.git
        path: '{{ .path.path }}'
        targetRevision: HEAD
        helm:
          valueFiles: [values.yaml, values.configmaps.yaml, values.extra.yaml]
          ignoreMissingValueFiles: true
      destination:
        server: https://kubernetes.default.svc
        namespace: dev
      syncPolicy:
        automated: { prune: true, selfHeal: true }

Onboard by adding environments/<env>/<app>/; the generator creates dev-<app>. Put CRDs, SecretStores, and Gateways in an infra ApplicationSet (or lower sync-wave) before app routes. Keep source.repoURL on the Application template—omitting it is not a valid Helm ApplicationSet source.

GitLab CI: shared include and concurrent updates

# Illustrative service-repo include — file:
# examples/gitops-repository-structure/ci/app-repo.gitlab-ci.yml
include:
  - project: 'platform/gitops-bridge'
    file: '/.ci/.gitlab-ci.yml'
    ref: main   # pin to tag/SHA in production (Illustrative hardening)

variables:
  IMAGE_BUILD_TOOL: buildkit
  IMAGE_UPDATE_METHOD: git_commit

Verified battle baseline: about five retries with git fetch + git rebase on push rejection, without sleep/jitter.

Illustrative hardening—bounded attempts, full jitter, fail-closed:

MAX_ATTEMPTS=5
BASE_SLEEP_SEC=2
for attempt in $(seq 1 "${MAX_ATTEMPTS}"); do
  if git push origin "HEAD:${GITOPS_BRANCH}"; then exit 0; fi
  git fetch origin "${GITOPS_BRANCH}"
  git rebase "origin/${GITOPS_BRANCH}"
  exp=$(( BASE_SLEEP_SEC * (1 << (attempt - 1)) ))
  [ "${exp}" -gt 30 ] && exp=30
  sleep_ms=$(( RANDOM % (exp * 1000 + 1) ))
  sleep "$(awk -v ms="${sleep_ms}" 'BEGIN { printf "%.3f", ms/1000 }')"
done
exit 1

Use the pattern above in the CI GitOps updater job. Optional: GitLab resource_group per hot values path to serialize concurrent writers.

Promotion metadata

• dev: push service dev → auto gitops_update into environments/dev/<app>
• stage: gated MR/job into environments/stage/<app> (Illustrative; battle used dev+prod)
• prod: manual GitLab job from main for the git_commit path

Battle evidence: six languages × two environments → twelve Applications Synced/Healthy; HTTPRoutes Accepted=True and ResolvedRefs=True. Manual prod rollback remained deferred—practice the runbook.

Gateway API ownership (kgateway)

Platform owns the shared Gateway (listeners, TLS certificateRefs, allowedRoutes). Apps own HTTPRoutes through chart route.*—exact hostnames, not wildcards. Cross-namespace attachment is gated by listener allowedRoutes (and matching sectionName), not by ReferenceGrant. Use ReferenceGrant only for cross-namespace backends (Service) or TLS Secret refs—grant lives in the target namespace.

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: external
  namespace: kgateway-system
spec:
  gatewayClassName: kgateway
  listeners:
  - name: https-wildcard
    protocol: HTTPS
    port: 443
    hostname: "*.example.com"
    tls:
      mode: Terminate
      certificateRefs: [{ name: wildcard-example-tls }]
    allowedRoutes:
      namespaces:
        from: Selector
        selector:
          matchLabels: { allow-external-gateway: "true" }

Manifests: examples/gitops-repository-structure/gateway/. During migration keep ingress-nginx and Gateway hostnames distinct—see ingress-coexistence.yaml and Ingress NGINX to kgateway. App wrappers must not enable production gatewayResource.

Secrets, RBAC, CODEOWNERS

• Secrets by reference only (envSecrets, External Secrets, Vault path refs). No PEM or passwords in Git.
• Separate AppProjects per environment; prod destinations allow-listed.
• CI deploy token writes bridge paths only; onboard MR token is different.
• CODEOWNERS: platform owns /argocd/, /environments/infra/, /.ci/; app teams co-own /environments/*/<app>/.

Details: examples/gitops-repository-structure/OWNERSHIP-CODEOWNERS.md.

Drift, rollback, pinning, disaster recovery

• Drift: selfHeal: true repairs toward Git. kubectl-edit is not durable.
• Rollback: git revert the bridge deploy commit or re-pin application.imageTag, then confirm Synced/Healthy and /health 200.
• Pins: exact chart version; prefer SHA/tag for CI include.ref and prod targetRevision during freezes.
• DR: restore Argo + cluster from IaC, sync last good bridge SHA, rematerialize Secrets, validate Gateway → HTTPRoutes → health, freeze image bumps.

Runbook: examples/gitops-repository-structure/PROMOTION-ROLLBACK.md.

Failure modes

• Unbounded GitOps push retries → bounded attempts + jitter (or resource_group).
• CI rewriting probes/routes with the image bump → mutate image fields only.
• Apps creating shared Gateways or embedding TLS keys → reject in review.
• Mixed chart pins after partial upgrade → inventory wrapper versions per ring.
• kubectl rollback under selfHeal → Git remains source of truth.

Official documentation

• Argo CD ApplicationSet
• Argo CD — Helm
• Helm dependency
• Kubernetes Gateway API / ReferenceGrant
• Universal Helm Chart / Artifact Hub

Why one Helm chart is enough → · One Helm chart vs many → · kgateway migration patterns → · Gateway API examples → · Ingress NGINX to kgateway →