Scenarios — Real Incidents & Interview Discussions
Six incidents told end to end: the symptom, the investigation, the mechanism underneath, and how to discuss each one in an interview.
Scenario: The Infinite Restart
A pod crash-looping with no logs. The triage order that finds the cause in ninety seconds, and why the JVM was the real culprit.
Scenario: The Error Nobody Could Reproduce
One in three requests failed. Both pods were healthy, from the same Deployment, running the same image — with different configuration.
Scenario: The Thundering Herd
A traffic spike triggered the autoscaler, which added twenty pods at once, which exhausted the database connection pool and took the site down.
Scenario: Three Replicas, One Disk
Valid YAML, accepted by the API server, and two pods stuck forever. What ReadWriteOnce actually means and why apply succeeding proves nothing.
Scenario: The Pod That Would Not Schedule
Pending with no events for ten minutes. How to read the scheduler's rejection message, and the four filters that eliminate every node.
Scenario: The Zero-Downtime Deploy That Wasn't
Every deploy produced a small burst of 502s. Readiness probes were correct, SIGTERM was handled, and requests were still being dropped.