On-Call That Does Not Burn the Team Out

The cost of on-call is not the incidents. Serious incidents are rare, and handling one is engaging work.

For a separate people-operations perspective, the detailed overview covers further reading on time management.

The cost is alerts that fire and require no action, the interruption of a night where nothing happened but you slept badly anyway, and the accumulated knowledge that any evening might be ruined. That is what makes people leave, and it is entirely fixable.

The number that predicts everything

Pages per shift that resulted in a corrective action.

Track two figures: how many times someone was paged, and how many of those needed a human to do something. The ratio is the health of the system.

A rotation where most pages need no action is broken, regardless of how few pages there are in total. It trains people to ignore alerts, which is what makes them miss the real one.

A rotation with more than one or two actionable pages per night is understaffed or has an unaddressed reliability problem, and no amount of process fixes that.

Also track: pages outside working hours, time to acknowledge, and how many pages went to someone who had to escalate because they could not act. That last one is a documentation and access problem, not a skill problem.

Fixing the alerts

Most on-call pain is alert quality, and it is the cheapest thing to fix.

Every alert must have an action. If the response is to look at it and go back to sleep, it should not page. Make it a ticket, or delete it.

Alert on symptoms, not causes. High CPU is not a problem if users are being served. Elevated error rate is a problem regardless of why.

Page on user impact; ticket everything else. The distinction is whether someone needs to be awake.

Delete alerts nobody acts on. Review the last quarter's pages, find the ones that never produced an action, and remove them. This single exercise typically halves page volume, and it is an afternoon's work.

Fix flapping. An alert firing and resolving repeatedly is noise. Add hysteresis, or a duration condition, or fix the underlying instability.

Deduplicate. One incident producing forty pages is one incident. Grouping is a feature of every alerting platform and it is frequently unconfigured.

Alert on leading indicators where you can — queue depth rising, pool approaching exhaustion — because acting before the outage is easier and faster than acting during one.

The structural things

Enough people in the rotation. Fewer than six means everyone is on-call frequently enough that it dominates their life. If you cannot reach six, that is a staffing fact worth naming rather than absorbing.

A secondary who is actually available. Not for escalation drama — so the primary can hand over when they are exhausted or when something needs two people.

Compensation or time back. Being available outside working hours is work, whether or not anything happens. Organisations that treat it as free get exactly the retention outcome they should expect.

Time off after a bad night. Explicit permission to not come in, granted by default rather than requested.

Handover between shifts. Ten minutes: what is ongoing, what is fragile, what changed. Prevents the incoming person discovering the situation at 2am.

Making incidents cheaper to handle

Runbooks linked from the alert itself. Not a wiki someone has to find at 3am. The alert should contain the link, and the runbook should start with what to check and what to do — not with an architecture overview.

Runbooks written for someone who did not build the system, and tested by having exactly that person follow one.

Access sorted in advance. The most common cause of a slow response is someone lacking permission at 3am. Verify that the on-call role can actually do what the runbooks require, before the shift.

Dashboards that answer "what is broken" in one screen. Not thirty panels — the handful of signals that discriminate between the likely causes.

A rollback that works and is fast. The single highest-value reliability investment for on-call quality, because most incidents follow a change, and reverting is the fastest resolution available.

A written escalation path with names and how to reach them, including who to wake for a database problem and who has authority to make a costly decision.

The feedback loop that matters

On-call load is a measurement of system reliability, and it only improves if the measurement reaches the people who can act on it.

Review pages weekly, not just incidents. Which alerts fired, which needed action, what would prevent each recurring.

The team that carries the pager should be the team that can fix the causes. Where operations carries the pager for software another team writes, the incentive to fix the noise sits with the people who never hear it, and the load never falls.

Turn recurring pages into work. A page that has fired six times is a defect with a measured cost, and it makes a stronger case for engineering time than most feature requests.

Include the noise in incident reviews. "This alert paged four times and needed nothing" is a finding worth an action. See incident review without looking for someone to blame.

During an incident

Acknowledge quickly, so nobody else is woken.

Stop the bleeding before diagnosing. Roll back, fail over, shed load. Understanding can come afterwards, and frequently understanding is much easier once the pressure is off.

Write in the channel as you go. It is the timeline for the review, it stops others repeating your steps, and it is how a second person joins without a briefing.

Escalate early. Escalating at twenty minutes is normal; escalating at three hours is a culture where asking for help is costly.

Hand over if you are impaired. Being awake for twenty hours makes people worse at exactly the reasoning an incident requires, and the handover costs less than the mistake.

The summary

Measure pages that needed action, not pages. The ratio tells you whether the rotation is healthy.

Delete alerts nobody acts on — the cheapest and largest single improvement available.

Six people minimum, a real secondary, compensation, and time back after a bad night.

The team carrying the pager must be able to fix the causes, or the load never falls.

An on-call rotation that is quiet is not luck. It is the accumulated result of treating every unnecessary page as a defect.