The Pager Never Sleeps: What Middle-of-the-Night Alerts Are Quietly Draining From Your Engineering Bench
Photo: Cory Huston, Public domain, via Wikimedia Commons
The alert arrives at 2:47 a.m. Eastern. A latency threshold has been breached in a service that, on closer inspection, was fine thirty seconds later. The engineer on call — senior, salaried, and now fully awake — resolves the page in four minutes and spends the next ninety minutes failing to fall back asleep.
Multiply that night by dozens across a year, across a team, across an organization, and you begin to see what the on-call rotation actually costs. It does not appear as a line item in any budget. It appears instead as slower feature delivery, rising sick days, exit interviews that mention 'sustainability' in careful language, and the gradual erosion of the institutional knowledge that keeps production systems running.
The Arithmetic of Alert Fatigue
Industry surveys have converged on a consistent picture for several years running. A majority of on-call engineers report that alerts interrupt their sleep at least occasionally, and a meaningful share report weekly sleep disruption. Meanwhile, studies of production alerting routinely find that a large fraction of pages — by some analyses, well over half — do not require immediate human action. Some are auto-resolved before the engineer finishes logging in. Others duplicate a page already acknowledged by a teammate.
The consequence is a phenomenon psychologists call the cry-wolf effect. When pages frequently turn out to be noise, engineers begin triaging their own alerts with decreasing urgency. The response time to the genuine 3 a.m. fire lengthens precisely because the pager has cried wolf too often. Incidents that might have been contained in minutes metastasize into customer-visible outages, which in turn generate more alerts, which deepens the fatigue. It is a feedback loop with a compounding interest rate.
The financial exposure is straightforward to sketch. Replacing a senior site reliability engineer in the current US market typically costs somewhere between half and a full year of that person's salary once recruiting fees, ramp-up time, and lost productivity are counted. If chronic on-call strain meaningfully contributes to even one avoidable departure per year on a mid-sized team, the organization has already spent more than it would on a year of serious investment in alerting quality, better runbooks, and rotation redesign — combined.
Where the Alerts Come From
Most pager load is not the product of malice; it is the residue of accumulation. Each team adds its own thresholds. Each incident generates a new alert intended to prevent recurrence. Nobody is tasked with subtracting. Within a few years, a monitoring stack built for a hundred services is paging for a thousand, and the signal-to-noise ratio has quietly inverted.
The remediation playbook is well understood among mature reliability organizations, even if unevenly adopted.
Anchor alerting to service-level objectives, not raw thresholds. A page should mean that a user-facing objective is actually at risk of violation — error rate, latency at the tail, availability — not that a CPU crossed an arbitrary line. Teams that migrate to SLO-based alerting commonly report reductions in page volume of thirty to fifty percent, with the surviving pages carrying far more informational value.
Audit the alert inventory ruthlessly. Every alert should have an owner, a documented response, and a recent history of being actionable. Alerts that fail that test get deleted or demoted to dashboard-level visibility. This is unglamorous work, but a quarterly alert audit is among the highest-leverage hours a platform team can spend.
Redesign the rotation itself. Rotations shorter than a week keep the psychological burden high; rotations longer than two weeks create fatigue accumulation. Most evidence favors one-week primary rotations with a secondary tier absorbing escalations. And the primary on-call engineer should have no other delivery commitments during their rotation — a standard many organizations still fail to meet.
Consider follow-the-sun staffing where scale permits. Organizations with distributed teams can hand the pager across time zones, so no individual bears the overnight burden routinely. It requires investment in documentation and handoff discipline, but it converts on-call from a hazing ritual into a sustainable operational function.
Retention Is the Real Metric
Engineering leaders tracking the health of their on-call culture should watch a small set of numbers: pages per shift, percentage of pages that were actionable, time-to-acknowledge trends, and — above all — the attrition rate of engineers with heavy on-call exposure versus those without. When that last number diverges, the organization is paying for its alerting design in its people.
The pager will never be popular. But it can be fair, rare, and respectful of the person carrying it. The teams that treat on-call as a designed product rather than an inherited burden are discovering that their best engineers stay longer, ship more, and — when the real 3 a.m. fire comes — answer it sharp instead of exhausted.