The Probabilistic Expert Test
Trust calibrated uncertainty over theatrical confidence
- Difficulty
- Moderate
- Time to result
- ~weeks to results
- Steps
- 6
- Confidence
- 96%
The Probabilistic Expert Test evaluates expertise by calibration rather than rhetorical confidence. A knowledgeable forecaster expresses uncertainty because they understand competing outcomes, imperfect data, and the limits of the model. The evaluator first translates the claim into a probability, then inspects the evidence, assumptions, alternatives, and stated uncertainties. A single surprising outcome does not invalidate a forecast: an event assigned a 30 percent probability should still occur roughly three times in ten. Quality therefore emerges across a series of comparable predictions, where predicted frequencies can be checked against actual results. The model counters a common bias in media and public life, where categorical speakers appear stronger than careful experts. Its output is a more defensible assessment of who is genuinely calibrated and who is merely certain.
Origin
Extracted from Masters in Business, where Michael Lewis explains why improving, data-based expertise can lose public trust as experts become more explicit about probabilities and uncertainty.
Core principles
- 01Uncertainty can signal deeper understanding
- 02A forecast is a probability, not a promise
- 03Outcomes alone cannot establish forecast quality
- 04Confidence and competence are different variables
- 05Expertise should be judged across repeated predictions
How to run it
- 1
Convert the claim into a probability
Replace vague confidence or categorical language with a numerical range when possible. Clarify the event, timeframe, and conditions.
Pro tip Ask what odds the expert would assign if forced to make the claim measurable.
Watch out Words such as likely and possible mean different things to different people.
- 2
Inspect the mechanism
Ask which evidence, model, causal process, or experience supports the estimate. Strong experts can explain how inputs affect the forecast.
Pro tip Request the factors most capable of moving the probability.
Watch out A precise number without a defensible mechanism may be false precision.
- 3
Look for acknowledged uncertainty
Identify assumptions, missing information, alternative explanations, and ways the forecast could fail. Appropriate doubt is evidence of scope awareness.
Pro tip Ask what would make the expert change their mind.
Watch out Do not confuse endless hedging with genuine calibration.
- 4
Evaluate a series, not one result
Collect multiple comparable forecasts and their eventual outcomes. Do not label a forecast wrong merely because a lower-probability event occurred.
Pro tip Group forecasts into probability bands such as 60–70 percent.
Watch out Outcome bias rewards lucky guesses and punishes sound estimates.
- 5
Check calibration
Compare stated probabilities with observed frequencies. Events forecast at roughly 70 percent should occur close to seven times out of ten over a sufficient sample.
Pro tip Track both calibration and the usefulness of the forecast.
Watch out Tiny samples cannot establish reliable calibration.
- 6
Discount unsupported certainty
Reduce trust when a speaker repeatedly presents uncertain events as inevitable without evidence or caveats. Prefer experts whose confidence changes with the quality of information.
Pro tip Separate presentation skill from forecasting skill.
Watch out Media visibility can systematically favor certainty over accuracy.
In the wild
A model gives one candidate a 70 percent chance of winning, but the other candidate wins. Instead of declaring the model useless, an evaluator reviews the model's assumptions and examines many elections assigned similar probabilities. If the favored candidates win close to seven times in ten, the model may be well calibrated despite this loss.
→ The forecast is judged by calibration rather than a single surprising result.
One commentator predicts a recession with total certainty, while another assigns a 40 percent probability and lists employment, credit, and inflation conditions that would change the estimate. A research team records both forecasters' claims over a year and compares their probability bands with outcomes.
→ The team identifies which forecaster is consistently calibrated instead of rewarding the more dramatic presentation.
Common mistakes
Treating probability as a promise
A 70 percent forecast still assigns a meaningful chance to the alternative. Calling the forecast wrong whenever the 30 percent event occurs misunderstands probability.
Rewarding confidence theater
Bold delivery can feel authoritative while providing no evidence of competence or calibration.
Judging from one prediction
A single outcome cannot distinguish skill from luck. Calibration requires a meaningful sequence of comparable forecasts.
Is it for you?
Best for
It is best for evaluating forecasts, pundits, advisers, medical claims, investment opinions, and other uncertain judgments.
Not ideal for
It is not ideal for deterministic questions with directly verifiable answers or urgent commands where probabilities cannot alter the required action.
From the transcript
“people don't really think probabilistically”
“we should respect experts who expressed doubts, even in their own understanding of the world, far more than experts who seem really confident.”
“they know little enough about what they're talking about to sound certain about it”
From the episode
Michael Lewis on His Journey to Wall Street