Kubernetes Secrets best practices with Helm

Keep credentials out of charts, Git, and CI logs. Ship secret references through Universal Helm Chart envSecrets, and let a store or controller materialize the Kubernetes Secret.

Threat model (summary)

Attackers do not need to “break encryption” on a Secret object if the plaintext already lives in Git, CI artifacts, helm get values, world-readable RBAC, or an unencrypted etcd backup. Full matrix: examples/kubernetes-secrets-helm/THREAT-MODEL.md.

• T01–T02: literals in values / Helm release metadata
• T03: over-broad secrets RBAC on controllers / break-glass (app SAs using envSecrets need no Secrets API verbs)
• T04: cross-namespace Gateway TLS refs without a Gateway→Secret ReferenceGrant
• T09: compromised Secret controller identity (ESO / Sealed) with wide secrets access
• T05–T06: failed rotation / restore reintroduces revoked keys
• T07–T08: CI debug logs; treating base64 as encryption

Base64 is not encryption

Kubernetes stores Secret data as base64 in the API object. Base64 is an encoding for binary-safe transport, not a confidentiality control. Anyone with get on that Secret (or an etcd snapshot without encryption at rest) can decode it. Official docs: Information security risks for Secrets and Encrypting Confidential Data at Rest.

# Demonstrates encoding only — do not store real credentials this way
printf 'not-a-secret' | base64
# bm90LWEtc2VjcmV0

Decision table: five delivery patterns

There is no universal winner. Match the pattern to who owns rotation and where ciphertext is allowed to live.

Expanded trade-offs: examples/kubernetes-secrets-helm/DECISION-TABLE.md. Sources: SOPS, Sealed Secrets, External Secrets, Secrets Store CSI Driver.

Helm pattern: references, not values

Universal Helm Chart exposes envSecrets: each entry becomes valueFrom.secretKeyRef. Values name the Secret and key; they never carry the credential. (Sibling key envFromSecrets exists for whole-key injection patterns—same rule: references only.)

# Excerpt — full file: examples/kubernetes-secrets-helm/values/app-envsecrets.yaml
envSecrets:
  enableEnv: true
  envs:
    - name: DATABASE_URL
      secretName: demo-api-secrets
      secretKey: DATABASE_URL
    - name: SESSION_KEY
      secretName: demo-api-secrets
      secretKey: SESSION_KEY

Rendered fragment (abbreviated from golden output):

env:
  - name: DATABASE_URL
    valueFrom:
      secretKeyRef:
        name: demo-api-secrets
        key: DATABASE_URL

GitOps repository boundaries

• App GitOps: chart version pin, public config, envSecrets / mount refs, HTTPRoute hostnames
• Secrets path or repo: SOPS/Sealed ciphertext or ExternalSecret CRs with stricter CODEOWNERS; no public mirrors of plaintext
• Platform: encryptionConfiguration, Gateway TLS Secrets, ReferenceGrant, ClusterSecretStore, audit policy

Promotion copies references and ciphertext blobs—not decoded passwords—between environments.

Least-privilege RBAC and workload identity

For chart envSecrets / secretKeyRef consumers: the pod ServiceAccount does not need Secrets API verbs. Kubelet projects Secret data into the pod; granting get/watch to the app SA expands blast radius if the pod is compromised. Default: no Secrets RBAC on app SAs.

Keep a named Role with resourceNames only for controllers, operators, or break-glass identities that call the Kubernetes API. Prefer cloud workload identity / vault roles over long-lived static tokens in Secret objects when the platform supports it.

# Controllers / break-glass only — NOT for envSecrets app SAs
# Full file: examples/kubernetes-secrets-helm/manifests/rbac-app-readonly-own-secret.yaml
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    resourceNames: ["demo-api-secrets"]
    verbs: ["get", "watch"]

Kubernetes RBAC docs: Using RBAC Authorization.

Encryption at rest, audit, backup

• Enable EncryptionConfiguration with a KMS provider for secrets.
• Residual risk: encryption at rest protects etcd/disk snapshots; it does not protect against authorized get, kubelet projection into pods, process env/memory, or node compromise after mount.
• Audit get/watch/patch on Secrets for break-glass identities and controllers.
• Treat etcd/Velero backups as secret material; after restore, force-rotate high-value credentials (see runbook).

kgateway / Gateway API: TLS refs and ReferenceGrant

TLS termination usually lives on the platform Gateway listener via certificateRefs to a Secret provisioned out of band. Application charts set route.gateway / route.gatewayNamespace / route.sectionName.

# Platform Gateway (illustrative) — Secret created out-of-band
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: external
  namespace: kgateway-system
spec:
  gatewayClassName: kgateway
  listeners:
  - name: https
    protocol: HTTPS
    port: 443
    hostname: "*.example.com"
    tls:
      mode: Terminate
      certificateRefs:
      - kind: Secret
        name: wildcard-example-tls
    allowedRoutes:
      namespaces:
        from: Selector
        selector:
          matchLabels:
            allow-external-gateway: "true"

Same-namespace certificateRefs (Secret in kgateway-system with the Gateway) need no ReferenceGrant. Application charts only set route.*; they do not embed certificates.

Cross-namespace TLS: set certificateRefs[].namespace to the Secret’s namespace and place a ReferenceGrant in that Secret namespace with from.kind: Gateway (not HTTPRoute—an HTTPRoute grant does not authorize Gateway listener certificate refs). Spec: ReferenceGrant. Evidence pack (do not apply as-is):

# Cross-ns TLS — examples/kubernetes-secrets-helm/manifests/gateway-tls-referencegrant.yaml
certificateRefs:
- kind: Secret
  name: wildcard-example-tls
  namespace: platform-certs
---
apiVersion: gateway.networking.k8s.io/v1beta1
kind: ReferenceGrant
metadata:
  name: allow-kgateway-gateway-to-tls-secret
  namespace: platform-certs
spec:
  from:
  - group: gateway.networking.k8s.io
    kind: Gateway
    namespace: kgateway-system
  to:
  - group: ""
    kind: Secret
    name: wildcard-example-tls

kubectl describe gateway -n kgateway-system external
# Expect Accepted=True and Programmed=True
kubectl describe httproute -n apps demo-api
# Expect parents: Accepted=True, ResolvedRefs=True

Rotation and revocation runbook

1. Create the new credential in the authoritative store (never in Git); keep the old key valid until revoke.
2. Publish via ESO refresh, sealed/SOPS MR, or controlled native update.
3. Gate: wait for in-cluster Secret update (ExternalSecret Ready=True + Secret resourceVersion/checksum change) before any rollout.
4. Dual-key window if supported; then kubectl rollout restart consumers; verify health.
5. Only then revoke the old credential; audit failed auth.
6. Incident path: revoke first, rotate to a new value, wait for Secret sync, restart, scrub any plaintext that ever hit Git/CI.

Failure/rollback table: examples/kubernetes-secrets-helm/ROTATION-RUNBOOK.md.

Validation checklist

• helm lint / helm template against application 0.4.3 with values/app-envsecrets.yaml
• Rendered manifests use secretKeyRef only—no literal passwords in output
• Reject values that embed plaintext credentials in Git
• Illustrative manifests keep REPLACE_* / ENC[REPLACE_*] placeholders; ReferenceGrant present for cross-ns TLS

Failure modes

• Missing Secret → pod CreateContainerConfigError; fix store sync, do not paste into values
• ESO SecretSyncedError → RemoteRef / store RBAC / network
• Missing ReferenceGrant → cross-ns TLS/backend ref rejected
• GitOps reapplies old sealed ciphertext after revoke → update ciphertext and sync order
• Cluster restore → treat as credential incident; rotate

Official documentation

• Kubernetes Secrets · Encrypt data at rest
• Helm chart best practices — Secrets
• Gateway API ReferenceGrant
• External Secrets Operator · Sealed Secrets · SOPS · Secrets Store CSI
• Universal Helm Chart

Related reading

Production Helm charts checklist → · Internal Developer Platform → · GitOps repository structure → · One chart vs many → · kgateway migration patterns → · Deploy FastAPI (envSecrets) →