Kubernetes Control Plane Troubleshooting

Common failures

• OOM API Server killed under memory pressure—check node resources and request limits.
• Certs Expired or mismatched TLS between apiserver ↔ etcd or clients.
• etcd Unreachable or unhealthy—API returns errors or timeouts on writes.
• Port Another process bound to 6443 (or configured secure port).
• RBAC Misconfigured ClusterRoleBindings—symptoms look like “forbidden” for controllers or users.

Diagnosis flow

Key commands

kubectl get --raw /healthz
kubectl get --raw /livez
kubectl get --raw /readyz

# systemd control plane (many kubeadm clusters)
sudo journalctl -u kube-apiserver -f

# Pod-based control plane
kubectl logs -n kube-system kube-apiserver-<node-name> -f

crictl ps | grep kube-apiserver
ss -tlnp | grep 6443

Sample: failing API Server pod

kubectl describe pod -n kube-system kube-apiserver-control-plane often shows exit reason, OOM, mount failures, or bad flags.

Name:             kube-apiserver-control-plane
Namespace:        kube-system
...
State:            Waiting
  Reason:         CrashLoopBackOff
Last State:       Terminated
  Reason:         OOMKilled
  Exit Code:      137
...
Events:
  Warning  BackOff  12s (x15)  kubelet  Back-off restarting failed container kube-apiserver

Common failures

• Pods stuck in Pending with scheduling events.
• Scheduler static pod / deployment not running on control plane nodes.
• Control plane or worker resource pressure (CPU, memory, ephemeral storage).

Diagnosis

Use events and Pod details—scheduling failures are usually explicit in Events.

kubectl get events -A --sort-by=.lastTimestamp | tail -40
kubectl describe pod <pod> -n <ns>

kubectl get pods -n kube-system | grep scheduler
kubectl logs -n kube-system kube-scheduler-<node-name> --tail=100

Scenarios

• Resources “0/X nodes available: insufficient cpu/memory”—raise limits, add nodes, or reduce requests.
• Taints “didn’t tolerate taint”—add tolerations or remove taints where appropriate.
• Affinity node/pod affinity rules too strict—relax requiredDuringScheduling or fix labels.
• PDB PodDisruptionBudget blocking eviction—not a scheduler bug; check voluntary disruption constraints.

Common failures

• ReplicaSet not creating Pods or not matching Deployment generation.
• Deployment stuck progressing—ReplicaSets exist but readiness never converges.
• Jobs not completing—controller not creating Pods or stuck on backoff limits.

Diagnosis

Correlate Deployment → ReplicaSet → Pod graph with events. Controller-manager logs show sync errors, quota denials, or API errors.

kubectl describe deployment <name> -n <ns>
kubectl get rs -n <ns> -l app=<label>
kubectl logs -n kube-system kube-controller-manager-<node-name> --tail=200

Scenario: “1/3 replicas available”

1. Confirm three ReplicaSet Pods exist: kubectl get pods -l app=...
2. If Pods exist but not Ready, debug probes, mounts, and image pull—not controller-manager.
3. If fewer than three Pods, check RS spec, quota, and controller-manager logs for sync failures.
4. Check PDB and rollout strategy maxUnavailable—may throttle creation during surge.

Common failures

• Quorum loss Majority of members down—cluster read-only or API failures.
• Latency Slow fsync/disk—watch delays, timeouts.
• Disk Full or high usage—etcd raises alarms, compaction needed.
• Backup/restore Wrong member set, version skew, or corrupt snapshot.

Diagnosis (etcdctl)

Set endpoints and TLS flags to match your cluster. Example with TLS:

export ETCDCTL_API=3
export ETCDCTL_ENDPOINTS="https://127.0.0.1:2379"
export ETCDCTL_CACERT=/etc/kubernetes/pki/etcd/ca.crt
export ETCDCTL_CERT=/etc/kubernetes/pki/apiserver-etcd-client.crt
export ETCDCTL_KEY=/etc/kubernetes/pki/apiserver-etcd-client.key

etcdctl endpoint health
etcdctl endpoint status -w table
etcdctl alarm list

Key metrics (watch in monitoring)

• WAL fsync duration — sustained spikes imply storage or overload.
• Backend commit duration — high values correlate with API slowness.
• Member leader changes — network or resource instability.

Recovery: snapshot restore (high level)

1. Stop kube-apiserver (and often etcd members) per your runbook—avoid split-brain.
2. Restore snapshot with etcdctl snapshot restore into clean data dirs for members.
3. Reconcile static pod manifests or systemd units so all members start with consistent state.
4. Start etcd, verify endpoint health, then start API Server; validate with kubectl get nodes.

See etcd backup & restore for the dedicated walkthrough.

Decision tree: what’s broken?

Commands by component

Related pages

• Health probes (liveness, readiness, startup)
• Cluster administration
• etcd backup & restore

CKA tips (troubleshooting domain)

• Start from symptoms: get events, describe, then narrow to the failing component.
• Know the difference between /livez (running) and /readyz (accepting traffic)—useful under load or startup.
• For scheduling, quote the exact event reason in your answer (e.g. FailedScheduling message).
• Practice etcdctl with TLS env vars; quorum loss is a common exam narrative.
• Time-box: fix the smallest change that proves the theory (one flag, one taint, one binding).