Calibrated Probabilistic Forecasting
Turn uncertain predictions into measurable probabilities and improve through feedback
- Difficulty
- Moderate
- Time to result
- ~weeks to results
- Steps
- 6
- Confidence
- 98%
Calibrated probabilistic forecasting converts a prediction into a measurable judgment about a defined event. The forecaster specifies what will happen, by when, and with what probability, then records updates as evidence changes. When the event resolves, a scoring rule such as the Brier score compares the probability with reality. Repeated scoring reveals whether the forecaster is appropriately decisive, chronically overconfident, or too reluctant to move away from 50 percent. The mechanism depends on tight questions, honest uncertainty, regular updating, and a durable record. Instead of rewarding vivid stories or credentials, it rewards estimates that repeatedly align with outcomes. Practice in forecasting tournaments creates the quick, clear feedback needed to turn these principles into judgment skills.
Origin
Extracted from Masters in Business, where Philip Tetlock described research conducted through the Good Judgment Project and intelligence-community forecasting tournaments.
Core principles
- 01Express uncertainty numerically rather than through vague language
- 02Define outcomes and deadlines before making a forecast
- 03Minimize the gap between assigned probabilities and observed outcomes
- 04Update estimates when diagnostic evidence changes
- 05Treat accuracy as a skill improved through repeated feedback
How to run it
- 1
Define a resolvable question
Describe one observable event and establish an explicit deadline and resolution standard. Remove language that could support multiple interpretations after the fact.
Pro tip Use a binary outcome when possible, even though the probability assigned to it can be granular.
Watch out A prediction without a deadline can remain unfalsifiable indefinitely.
- 2
Establish an initial probability
Translate uncertainty into a numerical estimate rather than words such as possible or likely. Use only as much precision as the available evidence can support.
Pro tip Start from a relevant base rate before considering case-specific evidence.
Watch out Do not confuse extra decimal places with genuine knowledge.
- 3
Record the rationale
Write down the evidence, assumptions, and causal reasoning supporting the estimate. This creates a record that can later distinguish sound reasoning from a lucky result.
Pro tip List evidence that would cause the probability to rise or fall.
Watch out A vivid narrative can feel persuasive without being diagnostically useful.
- 4
Update with evidence
Revisit the forecast as relevant information arrives and adjust it in proportion to the evidence. Avoid both constant reaction and stubborn refusal to change.
Pro tip Timestamp every update so the path of the forecast remains visible.
Watch out Do not rewrite the original rationale after learning the outcome.
- 5
Score the result
When the event resolves, compare the assigned probability with the binary outcome using a consistent scoring rule. Aggregate scores across many forecasts rather than judging skill from one prediction.
Pro tip Use a Brier score or another proper scoring rule that rewards both calibration and justified decisiveness.
Watch out One spectacular success or failure is not a reliable measure of forecasting skill.
- 6
Diagnose and retrain
Review scores for patterns such as overconfidence, excessive caution, or weak updating. Form a specific adjustment and test it across the next forecasting cycle.
Pro tip Practice in a tournament where questions resolve regularly and scores are visible.
Watch out Reading forecasting principles without repeatedly making and scoring forecasts produces limited improvement.
In the wild
An analyst defines the question as whether an official recession will begin before a specified date and assigns a 35 percent probability. Monthly indicators strengthen, so the analyst updates to 52 percent and records why. After the deadline, the result is scored and added to the analyst's long-term calibration record instead of being defended through vague language.
→ The analyst develops measurable evidence about forecast quality and learns how aggressively to update economic probabilities.
Instead of saying gold could surge, an investor forecasts a specific price threshold by a specific date with a 20 percent probability. The forecast is archived and evaluated at the deadline, preventing the investor from moving the goalposts later.
→ The market claim becomes falsifiable, scoreable, and useful for improving future judgment.
Common mistakes
Using elastic probability words
Terms such as distinct possibility can be reinterpreted after the outcome, protecting reputation while preventing learning.
Claiming false precision
Granular probabilities are valuable only when evidence and feedback support the distinctions being made.
Judging skill from one outcome
A low-probability event can occur and a high-probability event can fail, so performance must be evaluated across many forecasts.
Is it for you?
Best for
It is best for investors, analysts, policymakers, and teams making repeated decisions under uncertainty.
Not ideal for
It is not ideal for distant or poorly defined questions whose outcomes cannot be objectively resolved.
From the transcript
“You want to minimize the gaps between probability and reality.”
“The takeaways are that it is possible to make better probability estimates of events that many people thought it would be impossible to estimate probabilistically.”
“What it suggests is we'd be better off if we were aware of our limitations, achieved a certain baseline of appropriate humility, and got in…”
From the episode
Barry Ritholtz's Masters in Business: Philip Tetlock Interview
Philip Tetlock