Incidents & Postmortems

Something in production is broken, patients can’t book, doctors can’t see the queue, a page is down. What happens in the next ten minutes matters more than what happens in the next ten hours. This is how we run that window.

Declare early#

The failure mode is not “we called it an incident and it turned out to be minor.” It’s the opposite: someone spends forty minutes quietly debugging alone before anyone else knows production is down. Don’t hero it.

If in doubt, declare it. A false alarm costs a few minutes of attention. A silent outage costs trust and time, and both compound the longer no one else knows.
Declaring is not admitting failure. It’s the fastest way to get more eyes, more context, and permission to act decisively instead of tentatively.

One incident lead#

The moment an incident is declared, one person is the lead. Not a group, not “whoever’s around”, one name.

This mirrors the DRI principle: the lead is accountable for the incident being driven to resolution, not necessarily the one with their hands on the keyboard fixing it. Their job is to keep the response moving, decide what happens next, and make sure the right people are pulled in, not to personally diagnose the root cause while also trying to track status and communicate out. On a small team the lead may also be the one fixing it, but even then, someone is explicitly the lead so it’s never ambiguous who is making the call.

If the incident runs long enough to need a handoff (crossing a timezone boundary, a shift change), that handoff is explicit and written, the same discipline described in The Team: what’s been tried, what’s still unknown, what the next lead should do first.

Stabilize before you diagnose#

The instinct under pressure is to find out why before making it stop. Resist that. Get the system back to a known-good state first, understand it fully afterward.

Lever	Use when
Flip the flag off	The problem tracks to a specific flagged feature
Revert or roll back the deploy	The problem tracks to a recent change, flagged or not
Scale, restart, or fail over	The problem is load, a stuck process, or a dead dependency, not a code defect

This is the same lever table from Releasing & Rollback: reach for the most reversible option first. A flag flip is seconds and needs nobody; a rollback is clean precisely because our commits are small and green. Root-causing the actual defect can happen once patients can book again, it does not need to happen before.

Communicate on a cadence#

While the incident is open, people waiting on you should never have to ask “is anyone working on this.” Post a status at a fixed interval, even when the update is “still investigating, no change.” Silence during an incident reads as either nobody is working it or it’s worse than it is; neither is true, so don’t let the gap imply it.

A status update, in or out of a formal incident channel, covers three things: what’s known, what’s being tried, and when the next update lands.

Then, a blameless postmortem#

Once the system is stable, the incident isn’t closed, it’s written up. Blameless means the postmortem asks what about the system allowed this, never who caused it. A person made a reasonable decision with the information they had; the system let a reasonable decision produce an outage. Fix the system.

A postmortem is short and concrete:

Timeline. What happened, when, in the order it was discovered, not a reconstructed narrative that skips the confusion.
Root cause. The actual mechanism, traced with the help of Observability, not “human error” as a stopping point. Human error is where the investigation starts, not where it ends.
What we’re changing. A specific, systemic fix: a guardrail, a test, a flag, an alert, so the same failure can’t recur the same way. If the only output is “be more careful,” the postmortem isn’t finished.

Systeric keeps real postmortem examples on file for its clients, incidents are not treated as something to bury once resolved; they’re a record the next engineer can learn from.