Production Helm charts: a practical checklist

Walk this list against a packaged chart before you treat a release as production-ready. Each check has a pass fixture, a known failure, and a command you can rerun.

1. Chart.yaml and versioning

• Confirm apiVersion: v2, type: application, and SemVer version (chart) separate from appVersion (app).
• Bump chart version for every packaged change consumers depend on.
• Pin GitOps wrappers to an exact chart version (dependencies[].version: "0.4.3"), not a floating tag.
• Publish a changelog entry for breaking values renames before consumers upgrade.

# Chart.yaml (consumer wrapper)
dependencies:
  - name: application
    version: "0.4.3"
    repository: https://chaser100.github.io/u-helm-chart

2. values.schema.json

Treat the schema as a CI gate, not optional docs. Run it in CI with helm lint / helm template so invalid types fail before merge.

Pass: integer replicaCount, enum imagePullPolicy.

Fail (captured):

# fail-schema-replicaCount.yaml
replicaCount: "two"
# Error: Invalid type. Expected: integer, given: string

# fail-schema-imagePullPolicy.yaml
imagePullPolicy: Sometimes
# Error: must be one of "Always", "IfNotPresent", "Never"

3. Deterministic naming, selectors, labels

• Release name becomes app.kubernetes.io/instance; keep it stable across upgrades.
• Selector labels must stay immutable for the life of a Deployment (app.kubernetes.io/name + instance on this chart).
• Do not change nameOverride / fullnameOverride so that selectors rewrite on a live Deployment—Kubernetes rejects selector mutations.
• Use chart helper labels instead of hand-rolled maps that drift between Service and Pod template.

4. Immutable fields and upgrade hazards

• Treat Service clusterIP, PVC names, and Deployment selectors as sticky once applied.
• Changing service.type from ClusterIP to a type that needs recreation requires an explicit cutover plan.
• Job/CronJob renames create new objects; old Jobs may linger unless TTL or hooks clean them.
• CRDs installed from app charts create ownership problems. Prefer platform-owned CRDs outside the application release.

5. Secure pod and container defaults

The chart package default is empty securityContext: {}. Production values must set hardening explicitly:

podSecurityContext:
  seccompProfile:
    type: RuntimeDefault
securityContext:
  runAsNonRoot: true
  runAsUser: 10001
  runAsGroup: 10001
  allowPrivilegeEscalation: false
  readOnlyRootFilesystem: true
  capabilities:
    drop: ["ALL"]
serviceAccount:
  create: true
  automount: false

Fail mode: empty security contexts in prod namespaces that enforce Pod Security Standards either reject pods (restricted) or quietly accept weak defaults (baseline without org policy).

6. ServiceAccount and RBAC

• Create a dedicated ServiceAccount per release (serviceAccount.create: true).
• Set automount: false unless the workload must call the API.
• This chart manages the ServiceAccount object; Role/RoleBinding are not first-class keys on 0.4.3. Grant API access through platform RBAC or labelled extraManifests, least privilege only.

7. Probes, resources, disruption

• Define both livenessProbe and readinessProbe with realistic timeouts. Do not point liveness at a dependency that can flap independently of process health.
• Set resources.requests and limits; empty resources undermine scheduling fairness and HPA signals.
• PDB is not a first-class key on 0.4.3. Ship policy/v1 PodDisruptionBudget through extraManifests with selectors that match chart labels (see pass fixture).
• topologySpreadConstraints is not a chart key either. Document the intended fragment and apply it with a post-renderer (or wait for a chart key)—do not invent values.topologySpreadConstraints.

8. HPA

autoscaling:
  enabled: true
  minReplicas: 2
  maxReplicas: 6
  targetCPUUtilizationPercentage: 70

With HPA enabled, HPA owns scale—do not keep overriding replicaCount in GitOps as if it were the source of truth. Keep minReplicas ≥ PDB needs.

9. Hooks, Jobs, and CronJobs

• Use chart job / cronjob values instead of ad-hoc helm.sh/hook annotations unless you own hook weight and deletion policy.
• Hooks that create CRDs or cluster-scoped objects make rollback harder; keep them out of app charts.
• Set Job ttlSecondsAfterFinished so failed migrate Jobs do not clutter namespaces forever.

10. Secrets references (never literals)

envSecrets:
  enableEnv: true
  envs:
    - name: DATABASE_URL
      secretName: checklist-api-secrets
      secretKey: DATABASE_URL

Pass: rendered manifest contains secretKeyRef only (see envSecrets above).

Fail: embedding credentials in values or ConfigMaps. Review rendered YAML for sample literals such as password: and DATABASE_URL: postgres, and assert secretKeyRef is present. That is a fixture guard—not proof that arbitrary secrets are absent. Treat full secret hygiene as a review/policy gate.

11. Tests, lint, render, policy

Minimum local gate:

helm lint application-0.4.3.tgz -f fixtures/pass-production.yaml
helm template checklist-api application-0.4.3.tgz -f fixtures/pass-production.yaml -n apps

• Add helm unittest or golden diffs in CI when templates change.
• Run policy checks (Kyverno/OPA/Conftest) on rendered YAML for non-root, limits, and forbidden NodePort/hostPath.
• Do not ship deprecated APIs (extensions/v1beta1, networking.k8s.io/v1beta1 Ingress).

12. GitOps compatibility and rollback

• The chart must render fully from values files in git—no required --set for production.
• Avoid lookup in templates for prod paths; it breaks dry-run and GitOps diffs.
• Rollback: revert the Git commit (or Helm revision) that changed chart version/values, then confirm Deployment rollout and HTTPRoute status after reconcile.
• Uninstall: check PVCs, Jobs, and extraManifests objects that may orphan after release delete.

# Imperative rollback evidence (lab only)
helm rollback checklist-api 1 -n apps
# GitOps rollback: revert the values/chart pin commit and let the reconciler apply

13. Observability

• Enable serviceMonitor only when an in-cluster Prometheus Operator (or equivalent) exists.
• Expose a stable metrics path; scrape annotations alone are not a substitute for ServiceMonitor in Operator-managed clusters.
• Carry trace/log correlation through the standard app.kubernetes.io/* labels the chart helpers already emit.

14. Gateway API / kgateway checklist

Prefer simple chart route.* for portable single-host/path HTTPRoute fields; use route.spec when you need the full HTTPRoute.spec. Keep kgateway-specific policies in labelled escape hatches.

• Ownership: platform owns GatewayClass / Gateway listeners; the app release owns HTTPRoute via route.enabled.
• parentRefs: set route.gateway (required). Optional route.gatewayNamespace and route.sectionName pin the listener.
• Listener permissions: Gateway allowedRoutes must allow the app namespace; otherwise the route stays unattached.
• Status: require HTTPRoute Accepted and ResolvedRefs. For Gateway, also require Programmed when following kgateway lab patterns.
• ReferenceGrant: required when the app HTTPRoute references a backend Service in another namespace. It is not a chart values key—ship it as platform YAML (see escape-hatch example). Do not treat ReferenceGrant as a generic HTTPRoute→Secret grant.
• TLS: certificate Secret refs belong on the platform-owned Gateway listener, not on the application HTTPRoute. The app chart must not embed cert material.
• Timeouts: portable via route.timeouts.request / backendRequest on 0.4.3.
• Retries: not a route.* key—use kgateway TrafficPolicy via extraManifests and label networking.contract/kind: kgateway-specific.
• ingress-nginx migration: keep ingress.* and route.* dual-published only during cutover; remove Ingress after HTTPRoute status is healthy. See Ingress NGINX to kgateway.

route:
  enabled: true
  gateway: external
  gatewayNamespace: kgateway-system
  sectionName: https
  hostname: checklist-api.example.com
  path: /
  pathMatchType: PathPrefix
  timeouts:
    request: 75s
    backendRequest: 75s

Fail (captured): route.enabled: true without route.gateway → route.gateway is required when route.enabled=true.

15. Positive vs failing examples (summary)

Production gate (copy/paste)

1. Chart version pinned; changelog reviewed for breaking values.
2. helm lint and schema validation green.
3. Rendered manifests: non-root, drop ALL, read-only root (or a documented exception), probes, requests/limits.
4. Secrets only via Secret references / ExternalSecrets.
5. Service type ClusterIP; no NodePort, hostPath, or localhost assumptions.
6. Exposure: one of route.* or ingress.* (dual only during migration).
7. HTTPRoute parentRefs and Gateway listener allowlist verified; status Accepted/ResolvedRefs.
8. HPA/PDB coherent; topology strategy documented when it is not a first-class key.
9. GitOps dry-run/diff clean; rollback owner and revision known.
10. Observability scrape path exists when ServiceMonitor is enabled.

One chart vs many → · Gateway API vs Ingress → · Ingress NGINX to kgateway → · Browse examples →

Platform engineering with Helm → · Common Kubernetes deployment mistakes → · Kubernetes Secrets best practices → · One chart vs many → · Gateway API vs Ingress → · Ingress NGINX to kgateway → · Browse examples →