Evidence-Tested Investing Process
Test investment rules across cycles, then follow them without emotional overrides.
- Difficulty
- Advanced
- Time to result
- ~ongoing to results
- Steps
- 6
- Confidence
- 97%
Begin with an investment idea that has an economic rationale, such as avoiding extremely expensive stocks, and translate it into measurable rules. Test those rules across numerous market environments rather than relying on one favorable interval. The evaluation should include maximum drawdown, drawdown duration, benchmark-relative returns, and the frequency and magnitude of outperformance. If the evidence is sufficiently robust, establish a repeatable selection and rebalancing process. Its behavioral value is as important as its analytical value: predetermined rules reduce the temptation to abandon holdings during panic or chase them during exuberance. The methodology may be improved as better evidence emerges, but changes should result from research rather than reactions to current market conditions.
Origin
Extracted from Masters in Business, drawing on Jim O'Shaughnessy's quantitative research and his book What Works on Wall Street.
Core principles
- 01Prefer empirical evidence to compelling stories.
- 02Test ideas across many market cycles.
- 03Measure both returns and painful periods.
- 04Turn validated ideas into repeatable rules.
- 05Do not override the process during fear or exuberance.
How to run it
- 1
Form an Economic Hypothesis
Choose an investment idea with a plausible economic basis rather than starting with a fashionable data pattern. Define the variables and expected relationship clearly.
Pro tip Favor simple hypotheses that can be measured consistently over long histories.
Watch out A persuasive story is not evidence that the idea works.
- 2
Test Across Market Cycles
Evaluate the rule over as many bull markets, bear markets, recessions, and recoveries as the data permits. Avoid drawing conclusions from one convenient period.
Pro tip Use rolling test periods and rebalances to expose sensitivity to start and end dates.
Watch out A strategy can appear exceptional when measured only over a favorable sample.
- 3
Measure the Full Experience
Record maximum drawdown, drawdown duration, benchmark-relative performance, and rolling batting averages. Assess whether an investor could realistically endure the weak periods.
Pro tip Examine one-, three-, five-, and ten-year rolling results where possible.
Watch out Average return alone can conceal severe or prolonged losses.
- 4
Codify the Rules
Turn the validated idea into explicit criteria for selection, weighting, rebalancing, and removal. Make the rules clear enough to execute the same way repeatedly.
Pro tip Write the process before the next period of market stress.
Watch out Ambiguous rules create room for emotional exceptions.
- 5
Execute Without Overrides
Apply the process through both exuberant and fearful markets. Treat discomfort as expected unless new evidence invalidates the methodology.
Pro tip Automate or schedule rebalancing decisions where practical.
Watch out Overriding a model during volatility can erase the benefit of having tested it.
- 6
Improve Through Research
Periodically reassess assumptions, data quality, and factor definitions. Adopt changes only when broader evidence supports them.
Pro tip Separate scheduled research reviews from day-to-day portfolio execution.
Watch out Do not disguise a reaction to recent performance as methodological improvement.
In the wild
During the Great Financial Crisis, some quantitative investors overrode their models because extreme volatility made the prescribed positions emotionally difficult to hold. An investor following this framework would compare events with the strategy's tested drawdowns and continue executing unless the underlying evidence or implementation had genuinely failed.
→ The investor avoids converting temporary fear into an untested portfolio decision.
An investor proposes buying inexpensive companies. Instead of selecting whichever valuation ratio recently performed best, the investor tests multiple definitions over rolling periods, measures drawdowns and benchmark results, and writes fixed rebalancing rules before allocating capital.
→ A loose value idea becomes a measurable and repeatable investment process.
Common mistakes
Optimizing for One Historical Period
Selecting the best-looking metric from a single sample can mistake period-specific luck for a robust effect. Use rolling periods and multiple cycles instead.
Judging Only by Average Return
Returns without drawdown depth, duration, and benchmark comparisons provide an incomplete picture of the strategy an investor must endure.
Overriding Rules Under Stress
Abandoning a tested process during panic or exuberance reintroduces the behavioral biases the process was designed to neutralize.
Is it for you?
Best for
It is best for long-term investors who want a disciplined, evidence-based portfolio process.
Not ideal for
It is not ideal for investors unwilling to tolerate periods when a validated strategy trails its benchmark.
From the transcript
“It's essentially like a very long term study just looking at the evidence as opposed to stories.”
“We're turning it into a process that we run time and again and don't override.”
“following that process without making any additional emotional overrides, has proven itself to be quite effective”
From the episode
At the Money: Benefits of Quantitative Investing