mikepreston.org

The Error Budget Is a Confession

Nobody asks for 99.9%. Ask a room of stakeholders how reliable the service should be and the answer is 100%, delivered with the faint surprise of someone asked whether they'd like the bridge to stay up.

It's worth taking that request literally for a moment. A 100% target has an error budget of zero. Every change carries some risk, so a zero budget is a permanent change freeze: no releases, no migrations, no dependency updates. Nobody in the room wants that either. The request for 100% isn't a target. It's a wish, and it contradicts itself as soon as anyone tries to ship something.

The error budget is the instrument that turns the wish into a number. Agree on 99.9% over 28 days and you've agreed to buy about forty minutes of unreliability a month, and to stop spending once they're gone. That's the stated preference, written down and signed, which is already more honesty than most reliability targets get.

Then the budget runs out, and there's a launch on Thursday.

I've argued before that a freeze needs a signed policy, and that overriding it should be deliberate, recorded, and reviewed. All of that holds. This is about what happens when you go back and read the record.

One override is an exception. A regulatory deadline, a security fix, and a contract with teeth are all good reasons to spend through a freeze, and a policy that can't bend for them won't survive. Five overrides in six quarters, each for a feature launch, each individually justified, aren't exceptions. They're the policy. The organisation has told you, in the only currency that counts, that it values the launch date more than the forty minutes, and it has told you consistently.

That's the confession. The real reliability target is the one you enforce, and when you don't enforce it, the real target is whatever availability you happened to achieve over the quarters you overrode. It's usually lower than the number on the dashboard, and nobody wrote it down.

None of this means the overrides were wrong. Shipping the feature may well have been the right call every time. Revealed preference isn't a moral failing; it's information. The failure is keeping two targets, one on the SLO dashboard and one in the release calendar, and letting everyone believe the first while the second makes the decisions.

So there are two honest options, and either will do. Change the target to match the behaviour: if the organisation keeps choosing launches over the last few minutes of budget, set the SLO where the launches leave it, and tell the people who depend on the service. Or change the behaviour to match the target: honour the freeze, and let the launch slip. Just don't pick a third option where the number says 99.9 and the practice says something else.

The budget you enforce is the reliability you wanted. Everything else was a wish.