Common Kubernetes deployment mistakes

An incident-informed field guide for restoring service safely, proving the cause, and making the next deployment less surprising.

First five minutes: diagnostic decision path

kubectl -n apps get deploy,rs,pods,svc,endpointslices
kubectl -n apps get events --sort-by=.lastTimestamp | tail -40
kubectl -n apps rollout status deploy/web --timeout=2m
kubectl -n apps describe pod POD_NAME
kubectl -n apps logs POD_NAME --previous

Decision: Pending Pods usually mean scheduling, image pull, or admission. CrashLoopBackOff means inspect exit reason and previous logs. Ready Pods with no endpoints usually means selector or readiness mismatch. Endpoints with failed traffic often points to policy, DNS, port, or route. An HTTPRoute whose parent status is not Accepted is a gateway attachment or reference problem.

1. Probes and startup time

Symptom / root cause: restarts or zero ready endpoints after deploy. A liveness probe targets a dependency-bound endpoint, or readiness starts before the process can serve.

# Unsafe: kills a valid 90-second warm-up
livenessProbe: {httpGet: {path: /ready, port: 8080}, initialDelaySeconds: 5}
# Corrected: startup gates liveness; readiness reflects traffic readiness
startupProbe: {httpGet: {path: /healthz, port: 8080}, periodSeconds: 5, failureThreshold: 24}
livenessProbe: {httpGet: {path: /healthz, port: 8080}, periodSeconds: 10}
readinessProbe: {httpGet: {path: /ready, port: 8080}, periodSeconds: 5}

Detection: inspect kubectl describe pod events and ready replicas. Rollback: undo or pause the failing rollout; do not merely increase all delays. Prevention: measure cold-start p99 and test dependency loss separately from process health.

2. CPU, memory, and OOMKilled

Symptom / root cause: Pending Pods, throttled latency, or OOMKilled. Missing requests leave placement unpredictable; a memory limit below peak terminates the process.

# Unsafe
resources: {}
# Corrected (illustrative sizes — replace from observed load)
resources:
  requests: {cpu: 250m, memory: 256Mi}
  limits: {cpu: "1", memory: 512Mi}

Detection: kubectl describe pod (Last State/events) and kubectl top pod -n apps when metrics exist. Rollback: revert or raise a reviewed limit after checking node headroom. Prevention: set requests from observed steady-state, limits from tested peaks, and alert on restart rate or saturation.

3. Security contexts

Symptom / root cause: permission-denied exits or admission rejection. Images expecting root conflict with restricted policy.

# Unsafe
securityContext: {privileged: true}
# Corrected — container securityContext (spec.containers[].securityContext)
securityContext:
  allowPrivilegeEscalation: false
  readOnlyRootFilesystem: true
  capabilities: {drop: ["ALL"]}
# Pod-level (spec.template.spec.securityContext)
securityContext:
  runAsNonRoot: true
  seccompProfile: {type: RuntimeDefault}

Detection: read admission and container events; test with the same service account. Rollback: roll back; do not weaken policy without a time-bounded exception. Prevention: build images with runtime-UID-owned writable paths and validate manifests in CI.

4. Selectors, ports, and immutable fields

Symptom / root cause: Service has no EndpointSlices, or apply fails because a Deployment selector changed. Labels drift; spec.selector is immutable after create.

# Unsafe: selector matches no Pods
selector: {app: api}
# Corrected: shared label contract with the Pod template
selector: {app.kubernetes.io/name: web}
# Service targetPort must match an exposed Pod port.

Detection: compare kubectl get pods --show-labels and kubectl get endpointslices -l kubernetes.io/service-name=web. Rollback: restore old labels/selector; a planned selector migration needs a new Deployment and cutover. Prevention: centralize Helm labels and reject selector mutation in CI.

5. Rollout strategy, PDB, and topology

Symptom / root cause: a routine deploy removes all capacity. Defaults or a small replica count permit unavailability. A PodDisruptionBudget limits voluntary disruptions such as drains and evictions—it does not constrain Deployment rollingUpdate. Tie replica count, maxUnavailable, surge capacity, and PDB rounding together; minAvailable: 2 needs at least three replicas to tolerate one voluntary disruption.

# Risky for a single-replica API
strategy: {type: Recreate}
# Corrected after capacity review
strategy: {type: RollingUpdate, rollingUpdate: {maxUnavailable: 0, maxSurge: 1}}
---
# Applyable PDB (selector must match the workload template labels)
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: web
  namespace: apps
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app.kubernetes.io/name: web

Detection: watch ready replicas and kubectl rollout status; test drains and topology spread separately from rolling updates. Rollback: kubectl -n apps rollout undo deploy/web after confirming prior revision safety. Prevention: canary high-blast-radius changes and pause on sustained error-budget burn.

6. DNS and NetworkPolicy

Symptom / root cause: Ready Pods time out or cannot resolve names. Default-deny egress blocks DNS, or a ports-only exception permits DNS-port traffic to every destination.

# Unsafe: default deny with no DNS exception
policyTypes: [Egress]
# Corrected: complete egress item with scoped to + DNS ports
# Warning: DNS pod/namespace labels vary by distribution — confirm before apply.
egress:
- to:
  - namespaceSelector:
      matchLabels:
        kubernetes.io/metadata.name: kube-system
    podSelector:
      matchLabels:
        k8s-app: kube-dns
  ports:
  - {protocol: UDP, port: 53}
  - {protocol: TCP, port: 53}

Detection: kubectl -n apps exec POD -- getent hosts service.apps.svc.cluster.local and a scoped connection test. Rollback: restore the last reviewed policy, not allow-all. Prevention: policy CI tests DNS, dependencies, and telemetry egress.

7. Configuration drift and secrets

Symptom / root cause: Pods run stale settings, or a secret leaks into Git, logs, or shell history. ConfigMap updates do not refresh environment variables; manual edits fight GitOps.

# Unsafe: credential committed in a manifest
stringData: {DATABASE_PASSWORD: "replace-me"}
# Corrected: synchronized secret reference (never print values)
env:
- name: DATABASE_PASSWORD
  valueFrom: {secretKeyRef: {name: web-runtime, key: database-password}}

Detection: compare rendered Helm output, desired Git revision, and live object; use secret metadata only. Rollback: rotate exposed credentials and reconcile from the declared owner. Prevention: scanning, encryption, and immutable versioned references. Longer form: Kubernetes Secrets best practices.

8. Jobs and migrations

Symptom / root cause: a failed migration blocks deploys, or rollback restores code while schema stays new. Non-idempotent hooks make recovery ambiguous.

# Unsafe: destructive one-shot migration coupled to the same release
command: ["migrate", "--drop-legacy"]
# Corrected path: expand schema, deploy compatible code, contract later
kubectl -n apps wait --for=condition=complete job/web-migrate --timeout=10m

Detection: inspect Job conditions, logs, locks, and backup/restore proof. Rollback: halt the rollout and use tested data recovery—Helm undo does not reverse data. Prevention: idempotent migrations, explicit ownership, and a release gate.

9. Observability after the failure

Symptom / root cause: a 5xx spike has no attribution to app, dependency, or gateway; every restart pages. Detection: define availability and latency SLIs at the request boundary, with saturation and dependency signals. Page on sustained multi-window burn rate; ticket low-urgency restart or certificate signals. Recovery: preserve timestamps, revision, route status, and request samples before remediation. Prevention: dashboards link to owned runbooks and deployment annotations.

10. Gateway API / kgateway routes

Symptom / root cause: an HTTPRoute exists but parent conditions show Accepted=False or ResolvedRefs=False. Common causes: wrong parentRefs / sectionName, listener allowedRoutes restrictions, missing cross-namespace ReferenceGrant for a backend in another namespace, TLS attached to the wrong Gateway listener, Service-port mismatch, and retries or timeouts that amplify failure.

Check Accepted and ResolvedRefs on the relevant parent status entry, with observedGeneration current. Same-namespace Service backends do not need a ReferenceGrant; cross-namespace backend references do (grant lives in the backend namespace). Cross-namespace Gateway attachment is governed by the selected listener’s allowedRoutes and matching sectionName, not by inventing a grant for the Gateway itself. Downstream certificate selection and termination belong to the Gateway listener and hostname—not the HTTPRoute. backendRefs.port is the Service spec.ports[].port, not targetPort or the container port.

# Unsafe: assumes a parent/listener and wrong Service port
parentRefs: [{name: public, namespace: kgateway-system, sectionName: https}]
backendRefs: [{name: web, port: 80}]
# Corrected: match the platform Gateway listener and Service port contract
parentRefs: [{name: external, namespace: kgateway-system, sectionName: https}]
backendRefs: [{name: web, port: 8080, weight: 1}]

kubectl -n apps describe httproute web
kubectl -n kgateway-system describe gateway external
kubectl -n apps get svc web -o jsonpath='{.spec.ports[*].port}{"\n"}'
# Expect: Accepted=True and ResolvedRefs=True on the matching parent, observedGeneration current.

For retries and request timeouts under kgateway, use a reviewed TrafficPolicy (kgateway-specific; ship via labeled extraManifests when using the Universal Helm Chart). Portable chart timeouts map to route.timeouts only—do not invent route.retries. Target retries at safe, retryable failures and keep a bounded total timeout so retries cannot multiply blast radius.

Rollback: retain a known-good route and revert only changed route or policy objects; do not delete ingress-nginx before equivalent Gateway traffic is observed. Prevention: release-check parent status, cap retries, test TLS hostname and certificate selection on the Gateway listener, and create reviewed ReferenceGrants when backends cross namespaces.

Ingress-nginx coexistence during migration

Run both controllers only with distinct public addresses, host ownership, and a staged DNS cutover. Do not point two controllers at the same hostname or address and infer safety from whichever responds. Shift a measurable slice, compare status codes and latency, and retain legacy until DNS, TLS, routes, and rollback are proven. See kgateway migration patterns and Gateway API versus Ingress.

Release checklist: prevention is not recovery

• Before: render and schema-check manifests; verify requests, probes, labels, PDB versus replica capacity, policy, route parent status, and GitOps owner.
• During: observe SLI and error-budget posture, rollout readiness, and gateway conditions; pause on sustained burn.
• After: record revision, verify endpoints and a real request, and keep a tested rollback boundary.
• After an incident: write timeline and evidence, then create owned remediation with a due date—do not call a workaround a preventive control.

Explore deployment examples → · Gateway API examples →

Explore deployment examples → · Production Helm charts checklist → · Kubernetes Secrets best practices → · Gateway API examples → · Gateway API vs Ingress → · kgateway migration patterns →