Model-Robust Decision Test
Choose solutions that survive when your preferred model is wrong
- Difficulty
- Advanced
- Time to result
- ~months to results
- Steps
- 5
- Confidence
- 97%
The Model-Robust Decision Test shifts attention from whether a model appears correct to whether the resulting decision remains sound when the model is incomplete. Begin by identifying assumptions and plausible failure modes, then evaluate the proposed action across alternative explanations, datasets, and market states. A robust decision does not need to maximize the result in one forecast; it should preserve acceptable outcomes across many credible forecasts. This encourages diversification, margins of safety, optionality, and reversible actions. The method is especially important when managing other people's money or making decisions whose consequences are difficult to undo. Healthy skepticism becomes constructive because it informs design rather than merely criticizing the analysis.
Origin
Extracted from Masters in Business during Gerard O'Reilly's discussion of model uncertainty and fiduciary responsibility.
Core principles
- 01Every model is an incomplete description of reality
- 02Useful models can still produce fragile decisions
- 03Data sources deserve skepticism as well as models
- 04The downside of being wrong belongs inside the decision
- 05Robust solutions work across several plausible worlds
How to run it
- 1
Expose the assumptions
Write down the assumptions that connect the model's inputs to its conclusion. Include assumptions about data quality, stability, causality, and behavior.
Pro tip Ask what must remain true for the recommendation to work.
Watch out Hidden assumptions cannot be stress-tested.
- 2
Construct plausible failures
Describe realistic scenarios in which one or more assumptions fail. Focus on credible alternatives rather than unlimited hypothetical disasters.
Pro tip Include both gradual model drift and abrupt structural change.
Watch out Do not dismiss a scenario merely because it has not appeared in the recent sample.
- 3
Measure decision sensitivity
Estimate how much the outcome changes under each failure scenario. Identify assumptions whose failure produces unacceptable harm.
Pro tip Rank assumptions by consequence as well as likelihood.
Watch out Small errors in a highly sensitive variable can dominate the entire result.
- 4
Redesign for robustness
Add diversification, optionality, buffers, or reversibility so the decision does not depend on one forecast. Accept a lower theoretical optimum when it materially improves resilience.
Pro tip Look for changes that cap downside without eliminating the main upside.
Watch out Optimizing exclusively for the base case often creates hidden fragility.
- 5
Choose and monitor
Select the solution with acceptable performance across plausible scenarios. Track indicators that reveal whether assumptions are weakening.
Pro tip Define invalidation thresholds before implementation.
Watch out Robustness is not permanent; changing data can require a new decision.
In the wild
An investor believes a factor will earn a premium but acknowledges that the explanatory model may be incomplete. Instead of concentrating entirely in that factor, the investor diversifies across securities and expected-return drivers, controls trading costs, and maintains a long horizon.
→ The portfolio can benefit if the model is useful without becoming catastrophic if the model disappoints.
A company forecast suggests demand will double, but management tests lower-growth scenarios before committing capital. It leases flexible capacity and stages hiring behind measured demand thresholds instead of building the full operation immediately.
→ The company preserves expansion upside while limiting losses from a forecasting error.
Common mistakes
Confusing explanatory fit with truth
A model can explain historical data well and still omit important mechanisms or fail after conditions change.
Testing only the base case
A solution optimized for the expected scenario may be dangerously sensitive to small assumption errors.
Using skepticism as paralysis
The purpose is not to prove that every model is flawed; it is to design an action that remains useful despite those flaws.
Is it for you?
Best for
It is best for investing, strategy, policy, and other decisions where uncertainty cannot be eliminated.
Not ideal for
It is not ideal when outcomes are easily reversible and experimentation is cheaper than extensive robustness analysis.
From the transcript
“All models are incomplete, none of them are true, none of them are perfect descriptions of reality.”
“if you ever believe that the model is reality, you're probably going to build non robust solutions and do them a disservice.”
“what if I'm wrong, do I still have a good in solution even if this model it turns out to be incorrect?”
From the episode
Gerard O’Reilly on Academic Research and Stocks