The Backtest Pain Test
Trust a strategy only when its edge comes with a credible source of pain
- Difficulty
- Easy
- Time to result
- ~days to results
- Steps
- 6
- Confidence
- 97%
The Backtest Pain Test starts from distrust: hypothetical and live past performance can both mislead. Instead of asking whether a chart looks attractive, reconstruct the strategy's process and determine why it should generate returns. Then search for the corresponding pain—tracking error, drawdowns, career risk, tail exposure, crowding, or years of underperformance—that prevents investors from easily arbitraging the edge away. An unusually smooth or spectacular backtest is less credible when no economic trade-off explains its persistence. Finally, inspect incentives. Evidence created by a firm selling the strategy deserves a steep discount, while independent and peer-reviewed research can provide stronger, though still imperfect, corroboration. The output is not certainty but a disciplined credibility judgment grounded in mechanism, costs, risks, persistence, and source bias.
Origin
Extracted from Masters in Business during Wes Gray's discussion of overfitting, strategy pain, and biased investment research.
Core principles
- 01Treat all past performance as evidence, not proof
- 02Demand a causal explanation for why the process worked
- 03Require a credible reason the edge can persist
- 04Look for painful trade-offs that discourage arbitrage
- 05Discount research produced by a product seller
- 06Seek independent evidence while recognizing every source's incentives
How to run it
- 1
Suspend belief in performance
Set aside cumulative returns, Sharpe ratios, and polished charts. Begin with the assumption that historical results may reflect overfitting, selection bias, or favorable conditions.
Pro tip Review the methodology before looking closely at the return chart.
Watch out Live performance is not automatically reliable merely because it is not hypothetical.
- 2
Reconstruct the process
Identify the inputs, portfolio rules, rebalance logic, constraints, and economic exposure. State plainly what the strategy does to earn its return.
Pro tip If the process cannot be explained clearly, pause the evaluation.
Watch out A complicated model is not necessarily a defensible model.
- 3
Explain historical success
Determine whether the results follow from behavioral, structural, risk-based, or informational mechanisms. The explanation should connect the process to the observed return pattern.
Pro tip Prefer mechanisms supported across markets or independent datasets.
Watch out A narrative invented after seeing the results may simply disguise overfitting.
- 4
Find the pain
Search for the cost that makes the opportunity difficult to exploit, such as long underperformance, severe drawdowns, career risk, or unpleasant tail exposure. A persistent edge should usually demand something investors resist bearing.
Pro tip Ask what would make a rational allocator abandon the strategy at the worst time.
Watch out A strategy promising exceptional returns, minimal risk, and no meaningful drawback is presumptively incredible.
- 5
Stress the omitted risks
Evaluate tracking error, tail risk, capacity, crowding, implementation costs, taxes, and sensitivity to changed market conditions. Determine whether these risks were represented honestly in the test.
Pro tip Use scenarios that differ materially from the calibration period.
Watch out A backtest can be statistically accurate yet economically unusable.
- 6
Audit incentives and corroborate
Identify who produced the evidence and what they gain if it is believed. Discount seller-produced research and seek independent, academic, or peer-reviewed confirmation.
Pro tip Compare evidence from sources with different incentive structures.
Watch out Independent researchers also have publication and prestige incentives, so corroboration is stronger than blind trust.
In the wild
A manager presents a model earning 50 percent annually with negligible risk. The allocator temporarily grants that the simulation was calculated correctly, then asks why such an opportunity would remain available and what painful trade-off supports it. The manager cannot identify prolonged losses, capacity limits, unique information, or another credible barrier to arbitrage.
→ The allocator rejects the strategy despite its exceptional historical statistics.
A value strategy has broad historical support but can lag its benchmark for ten or twenty years. The evaluator treats this underperformance as economically important evidence: it creates career and behavioral pressure that prevents universal adoption. Independent research across long periods and markets further supports the mechanism, while the allocator still tests costs and implementation.
→ The backtest becomes more credible because its persistence has both evidence and a painful trade-off.
Common mistakes
Believing the prettiest chart
Investors select the smoothest historical result without determining whether it has a credible economic mechanism. Optimization can manufacture attractive charts that do not survive.
Treating pain as disproof
Long underperformance is assumed to invalidate a strategy even when bearing that discomfort may be the reason its premium persists. Pain must be understood, not automatically dismissed.
Ignoring the researcher's incentives
Seller-produced evidence is accepted at face value despite the producer benefiting directly from product adoption. Independent corroboration is necessary but should also be assessed for bias.
Is it for you?
Best for
It is best for investors, allocators, and researchers evaluating quantitative funds, factor strategies, or systematic trading models.
Not ideal for
It is not ideal as a substitute for technical replication, live operational due diligence, or detailed statistical validation.
From the transcript
“the best rule is just never trust any past performance, especially hypothetical, but even live past performance.”
“The reality is what you should understand is what is the process fundamentally, and then obviously why has this work and why will it continue…”
“don't believe any back tests, especially if it shows a great thing, unless it also shows why it's so bad.”
From the episode
At The Money: Finding Alpha via Unique ETF Strategies