The Problem
On-call at a small company usually isn’t a system. It’s a person. One engineer — often the best one, the one who knows where the bodies are buried — is effectively always on. Alerts fire constantly, most of them noise, so everyone has learned to ignore them, which means the one that matters gets ignored too. There’s no real rotation, no escalation path, and no recovery time. It works right up until that person takes a real vacation, or quits.
This is two problems wearing one coat. It’s a reliability problem: an alert everyone ignores is worse than no alert, and a single point of human failure is a single point of failure. And it’s a retention problem: nothing burns out a strong engineer faster than being permanently, silently responsible for production at 2 a.m.
The fix isn’t heroics or a bigger tool. It’s turning on-call into a system with four moving parts, each of which you can design deliberately.
The Approach
Four levers. They reinforce each other: a real rotation only works if the alerts are worth waking up for, and good alerts only stay good if there’s recovery time to keep tuning them.
A real rotation, not “everyone, always”
The first lever is the simplest and the most resisted: put on-call on a schedule, with named people and defined shifts, and take everyone else off.
- Minimum viable rotation is three or four people. Fewer than that and the rest between shifts is too short to matter. If you only have two or three engineers who can hold the pager, that’s your real problem — spread the operational knowledge before you spread the rotation.
- One-week shifts are the common default. Long enough to see patterns and own the follow-up, short enough to end. Some teams do shorter primary shifts to protect sleep; either works if it’s predictable.
- Predictable beats fair-in-theory. People can plan their lives around a schedule they can see three months out. They can’t plan around “whoever’s around.”
The goal of the rotation is that when you’re not on it, you’re genuinely off it. That’s the part that prevents burnout, and it’s the part small teams skip.
Alert quality: page only on user-facing and actionable
A rotation is only humane if being on-call is mostly uneventful. The lever that makes it uneventful is ruthless alert quality. The bar for a page — the kind that wakes someone — is two tests, both required:
- User-facing. Is a customer being hurt right now? If the symptom is “a background queue is 5% deeper than usual” and no user notices, that’s a business-hours ticket, not a 2 a.m. page.
- Actionable. Is there something the responder can actually do right now? An alert with no runbook and no action is just anxiety with a sound.
Everything that fails either test gets demoted: to a dashboard, to a daily digest, to a business-hours Slack channel. The number that tells you whether you’ve got this right is pages per person per week. Above two or three sustained, your alerts are too loud and your team is being trained to ignore them. The work of on-call, between incidents, is tuning that number down.
A severity ladder and an escalation that answers
When something does fire, the responder needs to know how hard to hit it, and needs a backup who will actually pick up.
- Three severity levels is enough. Sev1: customer-facing outage, page immediately, all hands if needed. Sev2: degraded but working, page during hours / next morning otherwise. Sev3: ticket. Write the definitions down so the 2 a.m. judgment is already made.
- Every rotation has a secondary. The primary doesn’t answer within the SLO — laptop dead, dead asleep, in a tunnel — and it escalates automatically to a named secondary, then to a lead. “Escalate to the person who’s always on” is not an escalation policy; it’s the burnout you’re trying to fix.
- Escalating is not failure. Make it explicit and cultural: waking up the secondary because you’re stuck is the system working, not the responder failing.
Recovery: comp time, no blame, keep tuning
The last lever is what makes the whole thing sustainable over months instead of collapsing back to the hero model.
- Compensate real off-hours work. If someone was up for two hours at 3 a.m., they start late, or take the time back. Unpaid, uncompensated night work is a debt you’re taking out against retention.
- Blameless everything. The postmortem asks what in the system let this happen, never who screwed up. The moment on-call feels like exposure to blame, your best people stop volunteering to respond.
- Feed incidents back into alert tuning. Every page is data. A recurring low-value page gets demoted; a missing alert that should have fired gets added. The rotation that tunes itself gets quieter over time; the one that doesn’t gets louder until people leave.
The Template
An on-call policy your team can actually run. Fill it in, publish it where everyone can find it, and revisit it quarterly.
Rotation
- Responders in the rotation: _____________ (target: 4+)
- Shift length: _______ (default: 1 week)
- Schedule published _______ months ahead
- Handoff ritual: ________________________ (what the outgoing responder passes on)
What pages (the bar)
- A page requires BOTH: user-facing impact AND a clear action
- Everything else routes to: ☐ dashboard ☐ daily digest ☐ business-hours channel
- Target pages per person per week: ______ (alert if sustained above 2-3)
Severity ladder
- Sev1 — customer-facing outage: page immediately, escalate to all-hands if unresolved in ____ min
- Sev2 — degraded / at-risk: page in hours; next morning off-hours
- Sev3 — non-urgent: ticket, next business day
Escalation
- Primary responder SLO to acknowledge: ______ minutes
- Auto-escalate to SECONDARY after: ______ minutes unacknowledged
- Then escalate to: ______________ (lead / manager)
- Secondary is a named person every shift: ☐ yes
Recovery
- Off-hours response is compensated by: ☐ time back ☐ late start ☐ ______
- Postmortems are blameless: ☐ yes, documented
- Every incident reviewed for an alert-tuning action: ☐ yes
Operating Notes
Watch two numbers, not one. Everyone tracks incident count. The number that predicts burnout is off-hours pages per person — the 2 a.m. wakeups, specifically. A team can absorb a lot of daytime pages and very few nighttime ones. If the nighttime number is climbing, fix it before someone quits, because by the time they quit it’s too late to fix cheaply.
Rotate the boring work too. The person who always writes the runbooks and the person who always takes the ugly shift are accumulating invisible debt. Rotate runbook authorship, rotate the holiday coverage, rotate the secondary. On-call fairness is mostly about the parts nobody volunteers for.
If you can’t staff a humane rotation yet, that’s the signal — not that on-call is broken, but that too few people can safely hold production. The fix is spreading operational knowledge: pairing on incidents, writing the runbooks, doing the game-day drills. A rotation is the reward for having enough people who can respond. Building that bench is the actual work, and it’s worth starting before the hero burns out.