Estimate, Don’t Measure
Treat financial model outputs as uncertain estimates, not hard facts
- Difficulty
- Moderate
- Time to result
- ~weeks to results
- Steps
- 6
- Confidence
- 96%
This mental model changes how quantitative outputs are interpreted. Financial models require numerical inputs, so analysts often use observable proxies such as volatility, beta, or recent liquidity. Those proxies make calculation possible but do not capture every form of risk, especially liability timing, permanent drawdowns, feedback loops, or disappearing liquidity. Calling the result an estimate keeps assumptions visible and discourages false confidence created by decimal precision. The practitioner examines the math, checks whether it warns against a decision, and then overlays judgment about factors the model omits. Historical and out-of-sample stress tests expose fragile assumptions. The output is not a rejection of modeling but a calibrated decision that combines systematic analysis with explicit uncertainty.
Origin
The discussion derives the rule from Peter Bernstein's history of modern financial theory and its limitations before and after major market crises. Extracted from Masters in Business.
Core principles
- 01Models simplify reality to make analysis possible
- 02Precision does not eliminate uncertainty
- 03Outputs depend on assumptions and available data
- 04Judgment should challenge rather than replace quantitative analysis
- 05Extreme conditions can invalidate normal-market relationships
How to run it
- 1
Define the Decision
Specify the allocation, investment, or forecast decision the model is meant to inform. Identify the real-world loss that matters rather than defaulting automatically to the model's risk proxy.
Pro tip Express risk in terms of failed objectives, liabilities, or unrecoverable drawdowns.
Watch out A convenient proxy may answer a different question from the one you actually face.
- 2
Expose the Simplifications
List the assumptions that make the model calculable, including chosen factors, normal relationships, and expected market liquidity.
Pro tip Ask what the model knowingly leaves out.
Watch out A clear number can conceal the fact that it originated as a practical simplification.
- 3
Audit the Inputs
Check whether the dataset spans different regimes, locations, and severe historical outcomes. Look for recent-data bias and inputs selected because they produce a preferred answer.
Pro tip Use out-of-sample and long-horizon evidence whenever available.
Watch out Models are only as useful as the data and assumptions supplied to them.
- 4
Stress the Failure Conditions
Test scenarios in which volatility spikes, liquidity disappears, correlations converge, or liabilities fall due during a drawdown.
Pro tip Include feedback effects in which the modeled strategy itself changes the market.
Watch out Normal-period relationships may fail together during a crisis.
- 5
Label the Output an Estimate
Present the result as a conditional estimate with a range, assumptions, and known omissions rather than as a hard measurement.
Pro tip State what evidence would cause you to revise the estimate.
Watch out Extra decimal places do not create extra knowledge.
- 6
Combine Math With Judgment
Use the model to challenge intuition and screen out poorly supported decisions, then apply informed judgment to risks the model cannot represent.
Pro tip When the model signals a very bad idea, pause and investigate before overriding it.
Watch out Neither instinct nor a model should receive total confidence.
In the wild
A pension manager sees acceptable long-run volatility in a portfolio but has a large payment due next year. She reframes risk as the chance of a drawdown immediately before that liability, tests historical crash periods, and models reduced liquidity during forced selling. She lowers the allocation to assets that could become difficult to sell and holds a larger liquid reserve.
→ The portfolio is aligned with the timing of its obligations rather than an abstract volatility score.
A lender's model uses only recent national house-price data and predicts minimal downside. The risk team adds Great Depression data, regional collapses, and scenarios in which liquidity contracts as prices fall. The revised estimate shows that the original confidence depended on a narrow period selected for benign results.
→ The lender adopts more conservative limits and reports a range instead of a single precise loss estimate.
Common mistakes
Equating the Proxy With Reality
Volatility, beta, and other calculable inputs represent selected dimensions of risk rather than risk in its entirety.
Using Only Convenient Data
Analysts can fall into the streetlight effect by studying only visible or recent observations while ignoring severe outcomes outside the sample.
Rejecting Models Entirely
The alternative to overconfidence is not pure intuition. Models provide a systematic way to test reasoning when their limits remain explicit.
Is it for you?
Best for
It is best for investors, analysts, and executives using quantitative models to make decisions under uncertainty.
Not ideal for
It is not a substitute for exact measurement where stable physical quantities and validated instruments genuinely exist.
From the transcript
“I think in finance, if we essue the word measure and replace it with the word estimate, we're much better off”
“your models are only good as good as the data you have”
“They aren't the kind of infallible guide that some people felt them to be.”
From the episode
John Authers, Christine Harper Discuss ‘Capital Ideas’
John Authers