Prediction Error Audit
Separate analytical mistakes from conclusions distorted by desired outcomes
- Difficulty
- Easy
- Time to result
- ~days to results
- Steps
- 6
- Confidence
- 96%
The Prediction Error Audit begins by distinguishing an incorrect outcome from a defective process. Reconstruct the forecast, its assumptions, and the evidence available when it was made. Then compare those assumptions with what actually happened and identify which causal links failed. The crucial diagnostic question is whether the forecast came from hard analysis or from a desire to see a preferred moral, political, or competitive outcome. If new information changes the mechanism, revise the prediction quickly and explicitly. The audit does not demand perfect foresight; avoiding every mistake would encourage safe, empty commentary. Instead, it produces a corrected model, a record of the update, and a practical guardrail against allowing hope, resentment, or partisan preference to masquerade as analysis.
Origin
Paul Krugman explained how he reviews failed economic predictions, including his election-night recession call and an unrealized Bush-era debt crisis. Extracted from Masters in Business.
Core principles
- 01Correct mistakes promptly
- 02Distinguish weak reasoning from bad luck
- 03Test whether emotion shaped the forecast
- 04Update conclusions when conditions change
- 05Accept error as a cost of making useful predictions
How to run it
- 1
Reconstruct the Forecast
Write down what was predicted, when it was expected, and the causal argument behind it.
Pro tip Use the original publication or notes rather than relying on memory.
Watch out Do not quietly replace the original claim with a weaker version.
- 2
Compare It with Reality
Identify precisely which predicted outcomes occurred, failed to occur, or remain unresolved.
Pro tip Separate direction, magnitude, and timing errors.
Watch out A delayed outcome should not automatically be declared correct.
- 3
Inspect the Assumptions
Trace the failed result back to assumptions about policy, behavior, incentives, or external conditions.
Pro tip Look for the earliest broken link in the causal chain.
Watch out Do not blame an unforeseeable event before checking ordinary analytical weaknesses.
- 4
Test for Motivated Reasoning
Ask whether a desire to reward allies or see opponents suffer influenced the judgment.
Pro tip Imagine making the same forecast with the political or personal identities reversed.
Watch out Strong moral conviction can coexist with weak predictive reasoning.
- 5
Correct the Record
State the revised conclusion and explain which new mechanism or evidence changed it.
Pro tip Update quickly while the original prediction is still visible.
Watch out A silent correction prevents others from learning from the mistake.
- 6
Create a Guardrail
Turn the diagnosed failure into a question, data check, or review step for future forecasts.
Pro tip Add the guardrail to a repeatable forecasting checklist.
Watch out Do not overcorrect by refusing to make meaningful predictions at all.
In the wild
Krugman initially predicted severe economic consequences following Trump’s election. Within days, he reconsidered the mechanism, recognized that larger deficits could provide fiscal stimulus, and retracted the original call rather than continuing to defend it.
→ The revised forecast better reflected the subsequent stimulus and economic trajectory.
A founder predicts a rival product will fail because its leadership appears unethical. When adoption grows, she reconstructs her forecast and realizes that moral disapproval replaced analysis of distribution and customer demand. She publishes a revised competitive assessment and adds a demand-evidence check to future reviews.
→ Future forecasts become less emotionally distorted and more operationally useful.
Common mistakes
Judging Only the Outcome
A sound probabilistic forecast can fail, while a lucky guess can succeed. Review the reasoning as well as the result.
Protecting Reputation Through Silence
Failing to record a correction preserves neither credibility nor learning because the original claim remains available to others.
Eliminating All Predictive Risk
Never being wrong may mean that forecasts are too vague or cautious to provide value.
Is it for you?
Best for
It is best for analysts, investors, journalists, and leaders who make consequential predictions in public or private.
Not ideal for
It is not ideal when no original reasoning or measurable prediction was recorded.
From the transcript
“if you never make a mistake, if you never get something wrong, then you're not taking enough risks.”
“was that actually grounded in hard thinking? Or was I letting my my desire to see bad things happen to bad people influence my judgment?”
From the episode
Paul Krugman on Challenging Zombie Ideas
Paul Krugman