What is Toil in SRE?
Toil is manual, repetitive, automatable work that scales linearly with system size, is reactive rather than proactive, and doesn't provide enduring value: operational busywork maintaining systems without improving them.
Defining Toil: The Five Characteristics
Google's SRE team defined toil in the SRE Book as work with five key characteristics. Manual: requires human execution; writing automation doesn't count as toil. Repetitive: performing the same task multiple times with identical patterns. Automatable: could be handled by machines with proper investment. Tactical, not strategic: reactive responses to current system state, not improvements. No enduring value: system returns to previous state after completion. Examples of toil include manually restarting periodic service crashes, running deployment scripts without CI/CD automation, manual infrastructure scaling, individual certificate rotation, cross-environment data copying for debugging, alert triage and routing that could be auto-classified, multi-server configuration updates, and scheduled manual data cleanup jobs.
What is NOT Toil and Why Toil is Dangerous
Not toil: post-incident postmortem writing (produces lasting improvements), investigating novel failure modes (builds understanding), building automation systems (prevents future toil), on-call problem-solving requiring genuine judgment, planning reliability improvement meetings. Toil is dangerous because it compounds over time: it scales proportionally with system growth, eliminates time for engineering improvements, causes engineer burnout through repetitive intellectually unchallenging work, and introduces human error risk compared to automated processes.
Google's 50% Rule and Measuring Toil
Google's SRE Book establishes that SRE teams should spend no more than 50% of their time on toil and operational work. Exceeding this threshold requires management intervention through hiring, engineering effort redirection, or system load reduction. Toil is measured through time tracking (categorizing hours as toil vs. engineering), ticket analysis (reviewing operational tickets for repetitive manual patterns), on-call analysis (examining incident data for manual automation opportunities), and per-service tracking (identifying highest-ROI automation targets). AI platforms automate investigation and decision-making during incidents (work traditional runbook automation cannot handle) through observability querying, signal correlation, and autonomous diagnosis.
What to remember
- 1Toil represents manual, repetitive, automatable operational work scaling with system size and producing no lasting value
- 2Google's 50% threshold establishes sustainable operational time limits; exceeding it requires intervention
- 3Toil compounds through time: scaling growth, engineering crowding, engineer attrition, and process risks
- 4Measurement through time tracking, ticket analysis, and on-call data enables reduction
- 5AI tools address hardest-to-automate investigative work and ongoing automation opportunity identification
Frequently asked questions
What is toil in SRE?
Manual, repetitive, automatable operational work scaling with system size and producing no lasting value: activities like service restarts, deployment scripts, and certificate rotation.
What is NOT toil?
Engineering work producing lasting improvements: postmortems, novel failure investigation, automation building, reliability design, and judgment-based on-call work.
What is Google's 50% rule for toil?
No more than 50% of an SRE's time should go to toil and operational work. Exceeding this is unsustainable.
How do I measure toil?
Time tracking, ticket analysis for repetitive patterns, on-call data examination, and per-service tracking identifying high-impact automation targets.
What's the difference between toil and useful operational work?
Toil is repetitive and automatable producing no lasting value; useful work builds knowledge, creates improvements, or requires human judgment.
How do I reduce toil on my team?
Measure it first, prioritize by frequency/duration/risk, build automation for painful items, invest in self-service tooling, use error budgets for justification.
Does AI eliminate toil?
AI targets hardest-to-automate investigative toil through autonomous observability querying, signal correlation, and diagnosis: work traditional automation cannot handle.
Who coined the term "toil" in SRE?
Google's SRE team formalized this in the 2016 SRE Book's "Eliminating Toil" chapter.
Is on-call considered toil?
On-call involving real problem-solving is not toil; repetitive script execution without lasting improvements represents toil.
What's an example of toil in software engineering?
Manual service restarts, cross-environment data copying, individual certificate rotation, manual deployment scripts, infrastructure scaling, disk space clearing.
See it in action. No slides.
NeuBird AI compresses incident investigation from hours to minutes: autonomous root cause analysis, with zero manual triage.