Working through an incident

Production is down. What you do in the next ten minutes matters more than what you know about the code.

Restore first, diagnose second

The instinct is to find the cause. The priority is to stop the harm.

1. is it actually broken, and for whom
2. what changed recently
3. can it be undone
4. undo it
5. diagnose calmly, afterwards

A rollback that takes two minutes beats a root cause analysis that takes forty while users are affected. The fix can wait; the outage cannot.

Something changed

Most incidents follow a change. Before theorising, check what shipped in the last few hours: deployments, configuration changes, feature flags, migrations, and third party providers.

If a deployment lines up with the start of the incident, that is your first suspect, regardless of how unrelated it looks.

Roles

In anything larger than a two person team, name them explicitly:

Incident lead: decides, and is not typing. Their job is coordination. Communicator: updates the status page and stakeholders so the engineers are not fielding questions. Engineers: investigate and fix.

The most common failure is five engineers independently investigating, changing things at once, and nobody knowing what has been tried. One person saying "stop, what have we tried, what is next" is worth more than another investigator.

Communicate before you know

"We are aware of errors on checkout, we are investigating, next update in 15
minutes."

Sent within minutes, and repeated on schedule even when there is nothing new. Silence is read as nobody noticing, and it generates the support load that distracts the people fixing it.

Change one thing at a time

Under pressure people change three things and the error goes away. Now nobody knows which worked, whether the other two broke something, or whether it will recur.

Write down what you change, in a shared channel, timestamped. That log is also most of the postmortem.

Afterwards

A blameless review, focused on what in the system allowed this rather than who did it. "Someone made a mistake" stops the investigation one step before anything useful.

The questions that lead somewhere: why was it possible to deploy this, why did nothing catch it before users did, why did it take 40 minutes to notice, why did the rollback take 25.

Each has an answer that can be changed. "Be more careful" does not.

The metrics

Time to detect and time to recover, both usually more improvable than the failure rate itself. A system that fails often and recovers in a minute is frequently better than one that fails rarely and is down for hours.