⌂ Home

Kubernetes Application Debugging

A practical flow from pod status to logs, exec, port-forward, and ephemeral debug containers—when to use each tool and how to fix common app issues.

Recommended debugging workflow

Start broad (cluster view of the workload), then narrow to one pod, then inspect process I/O and network from inside or via a local tunnel.

1. Pod status
kubectl get pods → 2. Events
kubectl describe pod → 3. Logs
kubectl logs → 4. Exec
kubectl exec → 5. Port-forward
kubectl port-forward → 6. Debug pod
kubectl debug

When to use which tool

kubectl get / describe Scheduling failures, image pull errors, probe failures, resource pressure, and recent Events on the pod.
kubectl logs App stack traces, startup errors, dependency timeouts, and anything the process prints to stdout/stderr.
kubectl exec Verify env vars, files on disk, DNS, listening ports, and connectivity to Services from inside the network namespace.
kubectl port-forward Reproduce HTTP/TCP behavior from your laptop without exposing the app through a Service or Ingress.
kubectl debug Distroless or minimal images with no shell, read-only root filesystems, or production pods you must not change in place.

Quick checklist — common application issues

  • Pod phase and Ready condition make sense for your workload.
  • Events show no repeated BackOff, probe failures, or quota denials.
  • Logs show the app reached “listening” or completed startup without fatal errors.
  • Required environment variables and mounted config exist (exec: env, ls).
  • Container port matches Service targetPort (exec: ss -tlnp or app docs).
  • From inside the pod, backend DNS names resolve and HTTP/TCP checks succeed.
  • If still stuck, reproduce with port-forward or attach an ephemeral debug container.

kubectl logs

Logs are the fastest signal for application-level failures. Default stream is the primary container; use -c for sidecars.

Basic — default container

kubectl logs <pod>

Shows stdout/stderr for the default or only container. Add -n <namespace> when not using the current context namespace.

Specific container

kubectl logs <pod> -c <container>

Use this when the pod has init containers, sidecars, or multiple app containers so you read the right process output.

Previous crashed container

kubectl logs <pod> --previous

After a restart, retrieves logs from the last terminated instance—essential for exit code 1/OOMKill post-mortems.

Follow live

kubectl logs <pod> -f

Streams new log lines like tail -f. Combine with -c for the correct container.

Tail last N lines

kubectl logs <pod> --tail=50

Limits output to the most recent lines; pair with grep locally for noisy apps.

Since a time window

kubectl logs <pod> --since=1h

Accepts durations (5m, 2h) or RFC3339 timestamps. Useful right after an incident window.

All pods with a label

kubectl logs -l app=web --all-containers

Aggregates logs from every pod matching the selector. Add --prefix to disambiguate pod/container names in the stream.

Timestamps

kubectl logs <pod> --timestamps

Prefixes each line with API server receive time—helps correlate with probes, HPA, or external incidents.

Patterns worth searching for

Errors Error, Exception, FATAL, panic

Resources OOM, OutOfMemory, cannot allocate

Network connection refused, timeout, ECONNREFUSED, no such host

Permissions permission denied, EACCES, read-only file system

kubectl exec

Run commands in the pod’s namespaces (network, mounts, env) to validate what the app actually sees at runtime.

Interactive shell

kubectl exec -it <pod> -- /bin/sh
# or, when available:
kubectl exec -it <pod> -- /bin/bash

-it allocates a TTY for interactive use. Prefer the shell that exists in your image.

Single non-interactive command

kubectl exec <pod> -- cat /etc/config/app.conf

Good for quick file reads without opening a full shell session.

Environment variables

kubectl exec <pod> -- env | sort

Confirms ConfigMap/Secret keys were projected as expected and catches typos in variable names.

Network check from inside

kubectl exec <pod> -- wget -qO- http://backend-svc:8080/health

Uses in-cluster DNS and Service VIPs. Substitute curl if your image includes it instead of wget.

Mounted secrets and configs

kubectl exec <pod> -- ls -la /var/secrets/

Verifies mount paths, symlinks, and file permissions for volumes.

Multi-container pod

kubectl exec -it <pod> -c sidecar -- /bin/sh

Targets a specific container by name when the pod defines more than one.

Common gotchas

  • Distroless images often have no shell—use kubectl debug with --target instead.
  • Read-only root filesystem breaks ad-hoc writes; test reads and use ephemeral debug containers for writable scratch space.

What to check inside

  • DNS resolution (nslookup / getent if available).
  • File permissions and expected paths for config and TLS material.
  • Running processes (ps if present).
  • Disk space (df) when crashes mention I/O or temp directories.

kubectl port-forward

Creates a secure tunnel from your machine to a Pod, Service, or Deployment—ideal for local curls and browser tests without changing cluster networking.

Forward to a pod

kubectl port-forward pod/<name> 8080:80

Local port 8080 maps to container port 80 on the named pod. Use the literal prefix pod/ followed by the pod name (for example pod/my-app-7d9f8).

Forward to a service

kubectl port-forward svc/<name> 8080:80

Targets the Service’s endpoints; useful when you want the same load-balancing behavior as in-cluster clients.

Forward via deployment

kubectl port-forward deployment/<name> 8080:80

Kubernetes selects a pod from the Deployment; handy when you do not know the current pod name.

Background on Unix shells

kubectl port-forward pod/my-app 8080:8080 &
curl -sS http://127.0.0.1:8080/health

Append & to keep the forward in the background; then hit localhost with your tools.

When to use

  • Smoke-test an app before publishing a Service or Ingress.
  • Attach debuggers or HTTP clients to a specific pod replica.
  • Reach metrics or admin ports that should not be public.

Common issues

  • Port already in use on your laptop—pick another local port (left side of the mapping).
  • Pod not running—forward fails until the pod is Ready enough to accept connections.
  • Wrong target port—right side must match the container’s listen port, not only the Service’s external port.

Ephemeral debug containers (kubectl debug)

Attaches a temporary container to an existing pod (or copies the pod) so you can use debugging tools without rebuilding the image.

Debug a running pod (sidecar-style)

kubectl debug <pod> -it --image=busybox --target=<container>

--target names the existing container whose namespaces (PID, network, mounts) the debug container shares.

Copy pod for a safer sandbox

kubectl debug <pod> -it --copy-to=debug-pod --image=busybox

Creates a new pod cloned from the original spec so you can experiment without touching production replicas.

Debug a node

kubectl debug node/<node-name> -it --image=ubuntu

Schedules a privileged-style debug pod on the node; host filesystem is typically under /host.

When to use

  • Minimal or distroless images with no interactive shell.
  • Read-only root filesystem where you cannot install troubleshooting packages.
  • Production pods you must not modify permanently—ephemeral containers attach without changing the Pod spec’s permanent container list in the same way as editing the deployment.

Useful debug images

  • busybox / alpine — small shells and basic utilities.
  • nicolaka/netshoot — networking toolbox (curl, tcpdump, dig, etc.).

Limitations

  • Ephemeral containers cannot be removed from a pod once added; they persist in the pod spec for audit/history.
  • Node debug pods mount the host under /host—treat them as highly sensitive and short-lived.

Common application issues

Each pattern below ties a symptom to concrete checks and a typical fix.

Missing environment variable

Symptom: Crash or log line like “config key not found” / “required env X”.

Diagnose:

kubectl exec <pod> -- env | sort
kubectl describe pod <pod>

Fix: Align Deployment env entries with ConfigMap/Secret keys; verify configMapKeyRef / secretKeyRef names and namespaces.

env:
  - name: APP_FEATURE_FLAG
    valueFrom:
      configMapKeyRef:
        name: app-config
        key: feature_flag

Wrong port (app vs Service)

Symptom: Service shows endpoints but connections fail or probes never succeed.

Diagnose:

kubectl exec <pod> -- sh -c 'ss -tlnp 2>/dev/null || netstat -tlnp 2>/dev/null'

Fix: Make containerPort, app listen port, and Service targetPort consistent.

ports:
  - containerPort: 3000
---
spec:
  ports:
    - port: 80
      targetPort: 3000

Readiness probe failing

Symptom: Pods stay Running but never Ready; Service has no endpoints.

Diagnose:

kubectl describe pod <pod>
kubectl logs <pod> --tail=100

Fix: Correct HTTP path/command, increase initialDelaySeconds, tune timeoutSeconds / periodSeconds so slow apps have time to become healthy.

readinessProbe:
  httpGet:
    path: /ready
    port: 8080
  initialDelaySeconds: 15
  timeoutSeconds: 2

Startup too slow — restarts during boot

Symptom: CrashLoopBackOff during long migrations or JVM warmup; liveness kills the container too early.

Fix: Add a startupProbe with a generous failure budget, or raise liveness initialDelaySeconds.

startupProbe:
  httpGet:
    path: /health/started
    port: 8080
  failureThreshold: 30
  periodSeconds: 10

File permission errors on mounts

Symptom: “permission denied” reading Secret/ConfigMap files.

Diagnose:

kubectl exec <pod> -- ls -la /path/to/mount

Fix: Set defaultMode on the volume so the process user can read the files (e.g. 0400 or 0440 as appropriate).

volumes:
  - name: secret-vol
    secret:
      secretName: app-tls
      defaultMode: 0400

Connection refused to a backend

Symptom: App logs show errors calling another service; exec tests fail.

Diagnose:

kubectl exec <pod> -- wget -qO- http://backend.backend-ns.svc.cluster.local:8080/

Fix: Use FQDN or correct short name when crossing namespaces; verify Service name, port, and that the backend pods are Ready.

Quick reference

Master table

Tool Syntax (examples) Best for
get / describe kubectl get pods, kubectl describe pod <pod> Phase, conditions, Events, probe messages, volumes mounted.
logs kubectl logs <pod> -f --tail=100 -c <c> App errors, stack traces, dependency failures, historical crashes via --previous.
exec kubectl exec -it <pod> -- /bin/sh Live environment, files, DNS, ports, in-cluster HTTP checks.
port-forward kubectl port-forward svc/web 8080:80 Local browser/curl against pod or Service without new Ingress rules.
debug kubectl debug <pod> -it --image=netshoot --target=<c> No-shell images, deep network tracing, node-level inspection.

logs vs exec vs port-forward vs debug

Approach You get… Tradeoff
logs What the app already printed. No interaction if the app logs little or crashes before logging.
exec Interactive inspection in the pod’s context. Requires a shell or binaries in the image (not true for distroless).
port-forward Your laptop as a client to cluster workloads. Tunnel dies when the command stops; not a replacement for Ingress.
debug Toolbox container sharing namespaces with production. Ephemeral containers stay in the API record; node debug is high privilege.

Related pages