Wrong-First Robustness Test
Assume an attractive result is wrong, then try to break it with better data
- Difficulty
- Moderate
- Time to result
- ~weeks to results
- Steps
- 6
- Confidence
- 94%
Begin by treating an unexpectedly strong result as probably wrong rather than immediately celebrating it. Reproduce the calculation, inspect the data set, and examine how the analysis was constructed. Then increase the data frequency, vary the measurement period, and remove returns explained by familiar factors. If unexplained performance remains, test whether it survives different interest-rate, growth, and recession regimes. The output is not certainty but a stronger distinction between a fragile backtest and a potentially durable edge. This process also forces the researcher to identify what is genuinely differentiated from inexpensive, commoditized factor exposure before turning an analysis into an investment product.
Origin
Sherman described using this skeptical validation sequence when DoubleLine investigated the Shiller-Barclays CAPE indexes after an initially surprising factor decomposition.
Core principles
- 01Treat surprising performance as a hypothesis, not a discovery
- 02Recalculate results before trusting them
- 03Increase data granularity to expose hidden weaknesses
- 04Test multiple periods and economic regimes
- 05Demand an edge beyond commoditized factors
How to run it
- 1
Assume the Result Is Wrong
Treat an unusually attractive result as a possible error. Write down the specific result and the ordinary factors that should explain it.
Pro tip A skeptical starting point reduces confirmation bias.
Watch out Do not reject the result merely because it is surprising.
- 2
Reproduce the Analysis
Run the calculation again and verify that the same result appears. Inspect the data set, transformations, and implementation rather than relying on a summary output.
Pro tip Have another analyst independently reproduce the result when possible.
Watch out Repeated output from the same flawed code is not independent confirmation.
- 3
Increase Data Granularity
Move from coarse observations to finer ones, such as monthly to daily data, and examine alternative time windows. Look for sensitivity to a small number of observations.
Pro tip Record how each methodological change affects the result.
Watch out Higher-frequency data can introduce noise and trading assumptions that require separate treatment.
- 4
Remove Familiar Explanations
Decompose returns into known factors and determine whether meaningful residual performance remains. Clarify what edge exists beyond exposures readily available elsewhere.
Pro tip Compare the proposed strategy with low-cost factor alternatives.
Watch out Do not label unexplained residual return as skill automatically.
- 5
Test Economic Regimes
Evaluate the strategy across rate, growth, and recessionary environments. Prefer behavior that remains reasonably consistent rather than depending on one regime.
Pro tip Define regimes before examining their results.
Watch out A small number of historical regimes limits statistical confidence.
- 6
Decide Whether the Edge Is Investable
Assess whether the surviving result can be implemented at realistic cost and combined with existing capabilities. Proceed only when the mechanism and differentiation are credible.
Pro tip Separate statistical significance from practical usefulness.
Watch out A robust backtest may still fail after fees, capacity limits, or operational constraints.
In the wild
DoubleLine initially viewed the indexes as another value product. After factor decomposition left unexplained return, Sherman assumed the result was wrong, reran the analysis, changed the data frequency, inspected the data set, and tested different rate, growth, and recessionary periods. The residual performance appeared relatively robust, supporting further product development.
→ A surprising result survived increasingly demanding checks and became the basis for an investable equity-plus-fixed-income product.
Common mistakes
Falling in Love With the First Backtest
An attractive first result encourages researchers to stop testing precisely when scrutiny should intensify.
Testing Only One Market Environment
A strategy that works only in one rate or growth regime can look durable until conditions change.
Confusing Factor Exposure With Edge
Returns available through commoditized factors should not be presented as unique investment skill.
Is it for you?
Best for
It is best for investors, analysts, and product teams evaluating a statistically promising strategy or model.
Not ideal for
It is not ideal when the available data is too sparse or unreliable to support meaningful robustness tests.
From the transcript
“However, you don't know me well enough, Barry the first thing, I say, that's gonna be wrong, Right, it's gotta be wrong,”
“And then the next step, after we see it again, go for monthly to daily, start looking at different time periods like go, now, give…”
“And we started going through regime changes and look at different rate environments and different growth and recessionary areas.”
From the episode
Jeffrey Sherman on Trading in Academics for Fund Management
Jeffrey Sherman