Theory-Beats-Theory Method
Replace weak models by testing better explanations against reality
- Difficulty
- Moderate
- Time to result
- ~weeks to results
- Steps
- 6
- Confidence
- 97%
The Theory-Beats-Theory Method treats models as successive approximations to reality. Finding a flaw is only the beginning: the critic must formulate a competing explanation that preserves what the old model explains while accounting for evidence it misses. Each theory should generate observable expectations, face the same data, and remain open to replication. The mechanism prevents debate from collapsing into stone-throwing or loyalty to established narratives. It also avoids demanding that an early model be discarded wholesale when a revised version can improve it. Lo describes progress as moving from version 1.0 to 1.1 and eventually 2.0. In practice, contradictory results become prompts for model development, while public challenge and independent replication strengthen rather than merely threaten credible research.
Origin
Andrew Lo articulated the rule while describing how his evidence against the random-walk hypothesis forced finance researchers to improve accepted market theories rather than merely attack them.
Core principles
- 01Criticism is incomplete without a better alternative
- 02Every theory is an approximation rather than final truth
- 03Data must arbitrate between competing narratives
- 04Progress comes through successive model improvements
- 05Replication turns controversy into useful evidence
How to run it
- 1
Define the incumbent theory
State the current model, its assumptions, and the observations it predicts. Separate its genuinely useful explanatory power from institutional attachment to it.
Pro tip Express predictions clearly enough that contradictory evidence can be recognized.
Watch out Do not caricature the incumbent theory merely to make replacement easier.
- 2
Find the unexplained evidence
Locate reproducible observations the existing model handles poorly. Check the data and methods from several angles before treating the anomaly as real.
Pro tip Use robustness checks that would expose a coding, sampling, or measurement error.
Watch out One surprising result is not automatically a new theory.
- 3
Build a better approximation
Propose a mechanism that explains both the established facts and the anomaly. Specify what new observations would support or undermine it.
Pro tip Revise the old model where possible instead of discarding everything it gets right.
Watch out A more complicated theory is not better unless it improves explanatory or predictive performance.
- 4
Run a fair comparison
Test the incumbent and competing theories against the same evidence. Evaluate fit, mechanism, robustness, and practical implications rather than rhetorical appeal.
Pro tip Include out-of-sample or later-period evidence when available.
Watch out Precision can disguise a model that is precisely wrong.
- 5
Invite challenge and replication
Expose the work to skeptical reviewers and make it possible for others to reproduce the result. Treat credible criticism as a way to identify the next revision.
Pro tip Record disagreements about assumptions separately from disagreements about data.
Watch out Defending identity or status instead of the model blocks adaptation.
- 6
Promote the best current version
Adopt the theory that better fits the evidence while keeping it open to replacement. Continue the cycle when new anomalies emerge.
Pro tip Label the model as a current approximation, not settled reality.
Watch out Success today does not make a theory immune to future evidence.
In the wild
Lo and Craig McKinlay presented weekly-return evidence showing short-term momentum. A discussant argued the numbers had to be wrong because they implied enormous trading profits. Researchers checked and replicated the findings, while statistical arbitrage was independently generating such profits in practice.
→ Replication shifted a controversial anomaly into evidence requiring an improved theory of market behavior.
Rather than declaring efficient-market theory useless, Lo retained its protections against investment mistakes and added psychology, neuroscience, evolution, and ecology to explain changing efficiency, emotion, and competition.
→ The replacement theory preserved useful insights while explaining observations the original formulation omitted.
Common mistakes
Only throwing stones
Pointing out a defect without proposing a more useful explanation leaves users with no model for action. Criticism should culminate in a testable alternative.
Rejecting data that threatens doctrine
A result should not be dismissed solely because its implications seem inconsistent with accepted theory. Verify the methods and evidence directly.
Confusing precision with truth
A mathematically elaborate model may still omit the dominant mechanism. Favor explanatory adequacy over decorative precision.
Is it for you?
Best for
Researchers, strategists, and decision-makers challenging an established explanation with contradictory evidence.
Not ideal for
Situations where no observable evidence can distinguish the proposed alternatives.
From the transcript
“It takes a theory to beat a theory.”
“In academia, it's not good enough to just throw stones at an idea. You've got to come up with a better one.”
“We really have approximations to reality, and we try to improve on those approximations one after the other.”
From the episode
Interview With Andrew Lo: Masters in Business (Audio)
Andrew Lo