Hypothesis-First Strategy Testing
Form a causal investment thesis before testing historical data
- Difficulty
- Advanced
- Time to result
- ~months to results
- Steps
- 6
- Confidence
- 99%
Hypothesis-First Strategy Testing applies the scientific method to quantitative investing. Researchers begin by articulating a plausible causal mechanism and specifying how it should generate an advantage. Only then do they examine historical data to test whether the predicted pattern appears. This ordering constrains data mining because the strategy is not repeatedly redesigned merely to improve its backtest. A credible process also acknowledges that historical relationships can change, tests robustness across assumptions and periods, and reserves data or time for out-of-sample validation. After launch, live results become the strongest check on whether the proposed mechanism survives implementation costs and changing markets. The method favors understandable, durable ideas over statistically impressive constructions with no prior rationale.
Origin
Rob Arnott contrasted Research Affiliates' hypothesis-led index research with endemic data mining in quantitative investing on Masters in Business.
Core principles
- 01Begin with a plausible mechanism
- 02Use historical data to test rather than invent the hypothesis
- 03Expect live conditions to differ from the past
- 04Treat a strong backtest as evidence, not proof
- 05Validate strategies with out-of-sample and live results
How to run it
- 1
State the hypothesis
Write a clear prediction and the mechanism expected to produce it before reviewing favorable historical outcomes.
Pro tip Make the statement specific enough that evidence could disprove it.
Watch out A vague thesis can be retrofitted to almost any result.
- 2
Predefine the test
Choose the dataset, benchmark, period, portfolio rules, and evaluation metrics before running the analysis.
Pro tip Record the initial specification so later changes remain visible.
Watch out Repeatedly changing rules after seeing results creates hidden overfitting.
- 3
Test the prediction
Use historical data to determine whether the hypothesized mechanism generated the expected results.
Pro tip Inspect performance, risk, turnover, capacity, and stability rather than headline return alone.
Watch out A good backtest does not establish causality or guarantee future performance.
- 4
Challenge robustness
Vary reasonable assumptions and test other periods or samples to see whether the result survives.
Pro tip Look for broad consistency rather than one optimized parameter value.
Watch out Fragile results may be artifacts of a particular sample.
- 5
Validate prospectively
Track the strategy out of sample and, when appropriate, in live implementation against its stated expectations.
Pro tip Define acceptable variance before observing live results.
Watch out Implementation costs and market changes can erase historical advantages.
- 6
Update without mining
Revisit the underlying mechanism when evidence changes, but do not continually tune the strategy merely to restore an attractive historical chart.
Pro tip Require an economic rationale for every methodological change.
Watch out Optimizing the backtest after disappointment repeats the original error.
In the wild
Researchers hypothesize that selecting companies by operating growth and weighting them by their dollar contribution to economic expansion should create effective growth exposure. They then test that predefined mechanism historically instead of searching blindly for whichever combination of variables produces the best chart.
→ The resulting evidence supports a strategy grounded in an understandable economic mechanism.
Arnott says the original RAFI backtest indicated roughly two percent of value added relative to cap-weighted value. Over the following twenty years, he reports live value added of roughly two to two and a half percent, providing prospective evidence beyond the original historical simulation.
→ Long-term live performance offers stronger validation than the launch backtest alone.
Common mistakes
Searching for a pretty backtest
Constructing rules by repeatedly searching historical data rewards chance relationships and produces exaggerated expectations.
Assuming the future repeats the past
Markets, technologies, regulations, and participants change, so historical relationships may weaken or disappear.
Promoting only successful tests
Showing only attractive backtests hides the volume of failed experiments and creates severe selection bias.
Is it for you?
Best for
It is best for researchers evaluating systematic investment rules, signals, and portfolio methodologies.
Not ideal for
It is not ideal for exploratory analysis whose purpose is discovery rather than validating a predefined deployable strategy.
From the transcript
“Scientific method means you start with a hypothesis and you only use the data to test the hypothesis.”
“Go back historically and ask what can I construct that's worked. We don't do that.”
“Many back tests that look great fail to perform in real life. The world changes.”
From the episode
Focusing on Growth (Not Market Cap)