Most cluster trouble announces itself in a small number of ways: a rollout that never finishes, a pod that keeps restarting, a node that cannot be emptied for maintenance, a service that falls over in sympathy with another. This guide takes those symptoms one at a time, names the cause that tends to sit behind each and suggests the first remedy worth trying. It follows the troubleshooting-heavy view of KubernetesOps found in the xOps School Ops Atlas page on the subject, where diagnosis carries the largest share of the CKA exam blueprint at 30 percent.
Symptom one: the rollout that hangs
What you see: a deployment sits half-updated and eventually reports that it exceeded its progress deadline. By default that verdict arrives after ten minutes (600 seconds).
Likely cause: the new pods cannot become ready. Either they are unschedulable because the capacity they request does not exist, or they start and fail their readiness check. The timeout is only the messenger.
First fix: look at pending pods and their events before touching the deployment. If requests exceed what any node can offer, the answer is smaller requests or more capacity, not a longer deadline. Raising the deadline hides the problem for a while and delays the alarm.
Symptom two: the restart loop that slows down
What you see: a container restarts, then waits, then restarts again with longer waits. The pause starts at ten seconds and climbs to a cap of five minutes (300 seconds).
Likely cause: the process exits on startup, usually because of a missing setting or an unreachable dependency, or a probe is killing a healthy process. The Atlas singles out one classic mistake: liveness probes that check external dependencies. When a database hiccups, every replica fails its liveness check at once, all are restarted together and a small incident becomes a cascading one.
First fix: keep liveness checks local to the process, and let readiness checks handle the question of whether dependencies are reachable. Then read the previous container’s logs rather than the current one, because the useful message is usually in the instance that just died.
Symptom three: the node that will not drain
What you see: during an upgrade you try to empty a node and the operation stalls, repeating rejected eviction requests.
Likely cause: a disruption budget that permits no pods to be taken down, so every eviction is refused with a rate-limit style response and the tool keeps retrying. A different pattern is more subtle. The Atlas notes that a server error on eviction can mean two budgets select the same pod, a configuration collision rather than a platform fault.
First fix: find budgets with zero allowed disruption and correct them, and check for overlapping selectors. Force-deleting the pod is the shortcut people reach for, and the sheet treats it as a last resort to be agreed with the workload owner, since it bypasses the very protection the budget provides. One more setting is worth knowing: the Atlas recommends letting unhealthy pods be evicted regardless of the budget, instead of the default behaviour that waits for health.
Symptom four: the pod that vanished without a budget check
What you see: pods are removed from a node even though a protective budget exists.
Likely cause: the node ran short of memory or disk and the kubelet began evicting. Eviction under node pressure ignores disruption budgets entirely, and the order is decided first by how far a pod’s usage exceeds its request, then by priority. Pods with no declared requests are the first casualties.
First fix: make requests realistic. A workload that asks for little and uses a lot is exactly the one the node will sacrifice, and a workload that asks for a lot and uses little wastes money. The Atlas offers a useful check on this, a usage-to-request ratio per workload, and suggests a policy that rejects deployments with no requests at all.
A related diagnosis: the cluster that fails after a quiet change
Some failures do not start with a symptom you can see. The sheet recounts a large social platform’s March 2023 outage during a version upgrade, in which a label disappeared from control-plane nodes and the networking components that were selected by it stopped working, so traffic between nodes failed. The removal had been announced. Recovery took about five hours from an etcd backup, and the company said its restore documentation had fallen behind.
The diagnostic habit it teaches is to read release notes for removals before each upgrade, search your own configuration for anything that depends on a thing being removed, and keep the restore procedure current. If you are thinking about how to structure that kind of preparation as a client engagement, a clear services roadmap for new clients is a decent model for sequencing it.
A short triage order
- Check whether pods are pending, restarting or evicted, since each points to a different layer.
- Compare requests with actual usage on the affected nodes.
- Inspect disruption budgets for zero-allowed or overlapping selectors.
- Review recent upgrades or add-on changes, including anything on the deprecation list.
- Confirm that admission webhooks, which fail closed by default, are not blocking system namespaces.
That last item deserves a footnote. Webhooks default to rejecting requests when they cannot be reached, so a broken one can stop even core components from being scheduled. Keep them away from the system namespace and the network plugin.
Turning diagnosis into prevention
Every fix above has a preventive twin: request hygiene, local liveness probes, budget reviews and a rehearsed restore. Treat repeated symptoms as inputs to a planning document in the spirit of a structured digital growth blueprint, with owners and dates against each item. A cluster that is diagnosed the same way every time becomes predictable, and predictability is what operations teams are really after.

