A short fable about a method that produces confident numbers and calls it rigor.
There’s an old story about an emperor who is sold a suit of clothes so fine that only the wise can see it. He parades through the streets wearing nothing, and the crowd — not wanting to appear foolish — praises the fabric. It takes a child to say the obvious: he is wearing nothing at all.
I want to tell a version of that story about Kanban’s work-amount prediction.
The method goes like this. Don’t estimate. Instead, sample the historical distribution of cycle time and throughput, run a Monte Carlo simulation, and announce the result as a probability: “80% chance of finishing 20–25 items by June 1.”
This is presented as more objective than estimation. More data-driven. More rigorous. The suit is woven from statistics, and only the sophisticated can see it.
Let’s look at what’s actually being worn.
The Premise, Stated Plainly
For the method to produce a meaningful probability, you need all of the following, simultaneously:
- Stationarity — the underlying distribution isn’t shifting.
- Verifiable stationarity — you can detect when it shifts, in time to matter.
- Homogeneous work — items are comparable, or the mix is stable.
- Enough samples in the current regime — to estimate a distribution at all.
- A regime long enough to act on — the forecast doesn’t expire before you use it.
- Practitioners who actually check 1–5 — rather than running the simulation and calling it done.
Each is hard. Together they are near-contradictory. And that’s before we notice that none of these requirements are visible in the output.
Failure One: Non-Stationarity Is the Norm, Not the Exception
The method assumes the distribution of cycle time is stable. It is not.
Teams change. People join and leave. The backlog mix shifts. A migration enters. A dependency breaks. The codebase ages. Priorities move.
The “distribution” is a moving target. Sampling it assumes it isn’t moving. It is.
This is not a subtle theoretical objection. It is the everyday condition of software work.
Failure Two: You Cannot See the Change
You can only estimate a distribution from samples. Any two samples differ. So when the new sample differs from the old, you cannot tell whether:
- the distribution intrinsically changed, or
- you got a different draw from the same distribution.
This is the change-point detection problem, and it has no clean solution. Control charts and change-point algorithms detect shifts after they have already invalidated the forecast. By the time you are confident the regime changed, the new regime is already old.
The method does not know when it has stopped describing reality. And neither, in practice, does the team.
Failure Three: The Window-Length Contradiction
The obvious fix is to choose a window. But the choice is impossible:
- Long window → enough samples → but you are averaging over regimes that no longer apply.
- Short window → reflects the current regime → but too few samples to estimate anything reliably, and dominated by noise.
There is no window length that escapes this. “Use a short window” does not solve non-stationarity. It only picks which way you want to be wrong.
Failure Four: Continuous Reforecasting Is a Confession
Kanban practitioners treat “reforecast continuously” as a virtue. Look at what it admits.
If the forecast must be redone every week — or every day — to stay valid, then the previous forecast was never a forecast. It was a snapshot of a distribution that was already moving.
Redoing the forecast is not rigor. It is re-describing the present and calling it the future. The fact that the method’s own users treat constant reforecasting as necessary is evidence they have noticed the method does not hold — and renamed the symptom as a feature.
Failure Five: The Disclaimer Contradicts the Output
Here is the sharpest one.
The same practitioner who says:
“80% chance of finishing 20–25 items by June 1.”
will also say:
“But forecasts are sometimes wrong — don’t over-trust it.”
These two claims are incompatible.
- If the 80% is real, then 1-in-5 failure is already priced in. No disclaimer is needed.
- If the disclaimer is needed, then the 80% is not a real probability. It is a rhetorical decoration on a guess.
You cannot have it both ways. Either the number is a calibrated probability — and you can be scored on it — or it isn’t, and you should stop calling it one.
The hedge reveals the number is not what it claims to be.
Failure Six: The Worst One — Silent Failure
A wrong point estimate is visibly wrong. Someone notices. The team adjusts.
A wrong probability looks fine. It is still a number between 0 and 1. It is still formatted as rigor. It is still delivered with a histogram and a confidence band.
The method can be badly miscalibrated for months — because the world moved and the distribution didn’t — and nothing in the output signals that the assumption broke.
That is the structural failure. Not that the method is sometimes wrong, but that it is systematically wrong in a way that mimics being right.
Failure Seven: Homogeneity Is Assumed, Not Maintained
The distribution is only meaningful if items are comparable, or if the mix of sizes is stable. Kanban’s answer is “split the work smaller.”
But:
- Much work genuinely cannot be split — a compliance audit, a data migration, a hardware integration.
- The split is a workflow discipline, not a property of the forecasting method.
- If the team doesn’t do it, the distribution is a mixture of incomparable things — apples and oranges averaged into a number that describes neither.
The method assumes a condition it does not enforce.
Failure Eight: No Causal Content
The distribution knows how long things took. It does not know why.
It cannot see that this item has a novel dependency, an unknown API, a regulatory gate, a first-time integration. It treats every item as a draw from the same urn — even when anyone looking at the item would immediately know it is different.
A human estimator, looking at the item, can say: “This is unlike anything we’ve done.” The historical distribution cannot. It has no mechanism to represent novelty, because novelty is precisely what is absent from history.
Failure Nine: The Premise Is Practically Unsatisfiable
Put the requirements together:
- Stationarity.
- Verifiable stationarity.
- Homogeneous work.
- Enough samples in the current regime.
- A regime long enough to act on.
- Practitioners who check all of the above.
Each is hard. Together they are near-contradictory. Stability is only visible over long windows, but long windows span regime changes. Change detection lags, so the new regime is old before you are confident in it.
For most software work, the premise fails. And the method proceeds anyway.
The Dodge: “It’s a Limitation of Induction”
When this critique is raised, the standard reply is:
“This is not a limitation of Kanban. It is a limitation of induction.”
This is true and irrelevant.
Different methods have different failure profiles under non-stationarity. Kanban chose a method whose failure mode is silent. A point estimate’s failure mode is loud — the estimate is visibly wrong, and people adjust.
Choosing a method with a silent failure mode, then hiding behind “all induction fails,” is not philosophy. It is an excuse for a bad design decision.
What the Honest Position Looks Like
The only defensible version of the Kanban forecasting claim is much weaker than the literature admits:
If your process is stable, and you verify that it is stable, and you re-verify continuously, and your work is homogeneous, and you have enough samples in the current regime, and you treat the output as a rough range rather than a probability — then historical data can be a useful input.
That is a narrow, conditional, easily-violated set of requirements. It is not a general replacement for estimation. It is not “objective” where estimation is “subjective.” It is a different guess, dressed in statistics, with worse failure visibility.
The Child’s Line
The emperor is not wearing a suit. He is wearing a number that looks like a suit.
The number is not a probability, because the premise that would make it one is almost never satisfied. The disclaimer that accompanies it is not a caveat — it is the load-bearing admission that the premise is not met. And the method’s failures are not loud enough to correct course, because a wrong probability does not look wrong.
Once you see this, you are left with two honest positions, and no third:
- Accept it. Admit that the output is not a probability, that the premise is almost never satisfied, that the number can be badly miscalibrated for months without anyone noticing — and use it anyway, knowing it is a guess in statistical clothing. That is a defensible choice, as long as it is stated honestly. What is not defensible is using it while claiming it is rigor.
- Throw it away. Conclude that a method whose validity conditions are almost never met, and whose failures are silent, is not worth the ceremony — and look for something that either fails visibly or makes fewer assumptions.
What you cannot do is keep the number, keep the disclaimer, and keep calling it objective. That is the position that requires the crowd in the street to praise the fabric they cannot see.
The point is not to replace one statistical method with another. The point is that you have to choose between honesty and pretense — between admitting the number is unreliable and giving up the claim to rigor, or dropping the number and looking for something better. Pretending that choice doesn’t exist is how the emperor keeps walking.
If you’ve run Monte Carlo on a distribution that had already moved — and only found out months later — you already know what this post is about.