The Backtest Cockroach Test
Expose hidden data mining by probing what researchers tried and discarded
- Difficulty
- Moderate
- Time to result
- ~weeks to results
- Steps
- 4
- Confidence
- 94%
The Backtest Cockroach Test uses a small disclosure to detect a much larger overfitting problem. When shown an impressive strategy, ask whether the researcher tried an additional plausible variable or specification. If the answer is that it was tested but did not work, the presented model was selected from a wider, usually undisclosed search. Like seeing one cockroach and inferring more behind the wall, that admission should lower confidence in the chosen backtest. The evaluator then investigates how many variants were attempted, adjusts for multiple testing, and seeks genuinely out-of-sample or live evidence. The goal is not to select the historically best model, but to identify a robust strategy whose performance has a reasonable chance of repeating.
Origin
Campbell Harvey described this diagnostic question while discussing his research on separating luck from skill in asset management on Masters in Business.
Core principles
- 01A strong backtest is not proof of repeatable skill.
- 02Discarded variables reveal the breadth of hidden experimentation.
- 03Testing many alternatives raises the probability of a chance result.
- 04The best historical specification may disappoint most in live trading.
- 05Research incentives influence which results are presented.
How to run it
- 1
Probe an alternative
Ask whether the researcher tested another plausible variable, parameter, or specification. Frame the question neutrally so the answer reveals the actual research process.
Pro tip Choose an alternative that a competent researcher would naturally have considered.
Watch out A single alternative is a diagnostic probe, not a complete audit.
- 2
Infer the hidden search
If the alternative was tried and discarded, investigate how many other unsuccessful versions preceded the published result. Treat the chosen model as one selection from that larger search.
Pro tip Request a complete research log rather than only the final specification.
Watch out Undisclosed experimentation can make ordinary chance look like exceptional performance.
- 3
Adjust for multiple testing
Raise the required evidence according to the number of variables, models, and strategies tested. Do not evaluate the final result as though it were the only hypothesis considered.
Pro tip Include inherited choices from prior academic papers in the effective number of tests.
Watch out Researchers can unknowingly delegate data mining by adopting previously selected variables.
- 4
Demand independent validation
Compare the result with untouched out-of-sample data or live performance. Prefer stable results across periods and specifications to the model with the highest backtested score.
Pro tip Freeze the model before evaluating it on new data.
Watch out Repeatedly revising the model after each out-of-sample failure turns the validation set into another training set.
In the wild
An investment team presents a product with an impressive backtest. The evaluator asks whether it tried variable X, and the team replies that it did but the variable did not work. That answer reveals that the final model emerged from a broader search than the presentation disclosed.
→ The evaluator lowers confidence and requests the full testing history and independent evidence.
Fisher Black challenged Harvey's early result as data-mined. Harvey had not personally mined the data, but a later 25-year out-of-sample test produced zero predictability. He concluded that the prior academic results he implemented may already have embedded the selection process.
→ The failed replication redirected Harvey's research toward data mining and false discoveries.
Common mistakes
Selecting the best backtest
The best historical result often reflects the most favorable noise and can be the specification most likely to disappoint in live trading.
Ignoring discarded trials
Evaluating only the winning specification understates how likely it was that some apparently strong result would emerge by chance.
Treating borrowed research as unselected
A researcher may perform no direct data mining yet inherit selected variables and methods from earlier published work.
Is it for you?
Best for
It is best for evaluating quantitative products, trading strategies, and empirical investment research.
Not ideal for
It is not designed to assess discretionary investments that have no systematic historical test.
From the transcript
“So as soon as I hear that answer, I know that there's been extensive data mining, and it is like it's like seeing a cockroach…”
“The best one is the one that's most likely to disappoint in in live treating.”
“Well you expect that out of a hundred, five would work just by random chance.”
From the episode
Campbell Harvey on the Future of Finance (Podcast)
Campbell Harvey