Persistent-Pervasive-Robust Evidence Test
Trust strategies only after they survive time, markets, and specifications
- Difficulty
- Moderate
- Time to result
- ~weeks to results
- Steps
- 7
- Confidence
- 94%
This evidence test evaluates whether an investment pattern is persistent through time, pervasive across countries and asset classes, and robust to alternative specifications. Researchers begin with modern out-of-sample evidence, which accumulates slowly, and supplement it with older historical records. They then challenge the result by changing markets, periods, definitions, and implementation assumptions. A strategy earns greater confidence only when the same basic pattern survives these variations. Historical depth is not accepted uncritically: data quality, investability, trading constraints, and structural change must all be examined. The method therefore avoids two extremes—believing a recent backtest too readily or dismissing all old evidence. Historical data strengthens a case, but it should not be the sole motivation for adopting a strategy.
Origin
Extracted from Masters in Business during Antti Ilmanen’s discussion of nineteenth-century investment data and systematic strategy research.
Core principles
- 01New out-of-sample evidence accumulates slowly.
- 02Long histories supplement rather than replace modern evidence.
- 03A credible pattern persists across centuries or regimes.
- 04A credible pattern appears across countries and asset classes.
- 05A credible pattern survives reasonable specification changes.
- 06Old data demands skepticism about quality and investability.
How to run it
- 1
Define the Pattern
State the strategy, signal, portfolio construction rule, and expected mechanism precisely enough to test repeatedly.
Pro tip Freeze the main hypothesis before exploring many variations.
Watch out A vague strategy can be redefined until almost any history looks favorable.
- 2
Test Persistence
Measure whether the pattern survives across different decades, centuries, and economic regimes.
Pro tip Report weak and losing periods rather than only aggregate performance.
Watch out A long sample is not persuasive if one short episode creates all the returns.
- 3
Test Pervasiveness
Apply the same logic across countries, asset classes, and related markets where the mechanism should operate.
Pro tip Explain beforehand where the pattern should and should not generalize.
Watch out Pooling unrelated markets can disguise inconsistent results.
- 4
Test Robustness
Vary reasonable signal definitions, portfolio cutoffs, holding periods, and cost assumptions.
Pro tip Look for a broad plateau of acceptable results instead of one optimal parameter.
Watch out Performance that disappears after a small specification change is fragile.
- 5
Audit Historical Credibility
Review source quality, survivorship, missing observations, and whether investors could have implemented the trade at the time.
Pro tip Separate data accuracy from practical investability; both matter.
Watch out Verified prices do not prove that a diversified strategy was executable.
- 6
Assess Structural Change
Identify institutions, regulations, technologies, or market structures that could invalidate older evidence.
Pro tip Give more weight to evidence generated under conditions resembling the intended deployment.
Watch out Two centuries of data do not guarantee that the mechanism remains intact.
- 7
Set Provisional Confidence
Combine modern evidence, historical breadth, robustness, and mechanism quality into a cautious decision.
Pro tip Use old data to supplement the case rather than create it.
Watch out Do not treat statistical recurrence as proof of a permanent law.
In the wild
A research team tests a valuation signal in US equities, then repeats it in other countries, industry-neutral portfolios, bonds, and older datasets. It varies valuation definitions and transaction-cost assumptions while checking whether the underlying behavioral or risk mechanism remains plausible.
→ The team adopts the strategy only after finding a broad, imperfect, but recurring pattern rather than one optimized backtest.
Common mistakes
Trusting One Golden Backtest
A strategy may look exceptional because its parameters were optimized for a single market and period.
Treating Old Data as Conclusive
Historical records can contain quality, investability, and structural-comparability problems. They should strengthen modern evidence, not substitute for it.
Demanding Identical Results Everywhere
Robust patterns can vary in magnitude across markets and periods. The test seeks consistent direction and mechanism, not perfect uniformity.
Is it for you?
Best for
It is best for researchers and allocators evaluating factor strategies, systematic rules, and historical anomalies.
Not ideal for
It is not ideal when only a short, isolated dataset exists or when the strategy cannot be defined consistently across markets.
From the transcript
“some strategies have worked persistently over different centuries, pervasively across different countries and asset classes and robust with different specifications.”
“we accrue out of sample new experience so slowly”
“I still like it as a supplementary evidence, not as main motivation for anything.”
From the episode
Antti Ilmanen on Expected Returns
Antti Ilmanen