Regime-Conditioned Indicator Testing
Test each signal inside the market state where it is supposed to work
- Difficulty
- Advanced
- Time to result
- ~months to results
- Steps
- 7
- Confidence
- 96%
Start by defining market states through a quantified trend model rather than subjective labels. Test a candidate indicator across the complete dataset, then split its results by regime. A signal that appears useless over the combined sample may be highly informative in a bear state and harmful in a bull state, or vice versa. Compare continuation, reversal, return, and risk outcomes within each state, using a long enough history to include multiple cycles. Retain signals only when their conditional behavior is robust and economically meaningful. Convert the findings into explicit operating rules that specify when the indicator is active, when it should be ignored, and what probability it implies. This produces contextual evidence rather than universal chart folklore.
Origin
Extracted from Masters in Business, where Jeff deGraaf described Renaissance Macro's quantified trend model and its practice of evaluating indicators separately in bullish and bearish states.
Core principles
- 01Indicator performance depends on market state
- 02Quantify regimes instead of assigning them by intuition
- 03Test signals over sufficiently long histories
- 04Reject indicators that lack conditional usefulness
- 05Translate validated signals into explicit probabilities
How to run it
- 1
Define the regimes
Create objective rules that classify observations into bullish, bearish, or another necessary state. Ensure the classification can be reproduced from historical data.
Pro tip Prefer simple, stable state definitions before experimenting with many subregimes.
Watch out Subjective regime labels make backtests impossible to reproduce.
- 2
State the signal hypothesis
Specify what the indicator is expected to predict and in which regime it should work. Define the horizon and outcome metric in advance.
Pro tip Write the hypothesis before inspecting conditional results.
Watch out Post-hoc hypotheses encourage data mining.
- 3
Build the full-sample baseline
Measure the indicator's performance across all observations. Use this as a comparison point rather than the final verdict.
Pro tip Include return, volatility, drawdown, and hit-rate measures.
Watch out An average result can conceal opposite effects in different regimes.
- 4
Split performance by state
Recalculate the same metrics within each quantified regime. Identify where the signal adds information and where it fails.
Pro tip Report sample sizes alongside every conditional result.
Watch out Small regime samples can produce impressive but unstable estimates.
- 5
Test robustness
Repeat the analysis over different periods, assets, and reasonable parameter variations. Reject relationships that depend on one episode or narrow setting.
Pro tip Use long datasets containing multiple bull and bear cycles.
Watch out Recent ten-year results may not represent the full range of market behavior.
- 6
Create operating rules
Specify the regime in which the signal is enabled, its expected probability, and when it must be ignored. Connect the rule to position sizing or risk limits.
Pro tip Make the rule simple enough that another analyst can execute it consistently.
Watch out A valid bear-market indicator can be destructive when carried unchanged into a bull market.
- 7
Retest as data accumulate
Monitor live outcomes and periodically rerun the historical analysis. Modify the rule only when evidence, not discomfort, warrants a change.
Pro tip Maintain a record of every rule revision and its reason.
Watch out Frequent tuning can convert a robust process into curve fitting.
In the wild
A research team evaluates an oversold oscillator and finds weak results over the entire market history. After dividing observations with an objective trend model, it discovers that the signal identifies useful reversals during one state but repeatedly fights momentum during the other. The team enables it only in the validated state and publishes the associated probabilities.
→ A misleading universal indicator becomes a narrower, testable, and more reliable decision rule.
Instead of judging breakouts from the most recent decade, analysts test them across a single-stock database extending to the early history of the S&P. They compare full-sample performance with behavior under different trend conditions and report whether breakouts work universally, conditionally, or not at all.
→ The research distinguishes enduring conditional evidence from recent-market anecdotes.
Common mistakes
Testing only the aggregate sample
Combining bullish and bearish observations can average away a signal that has strong but opposing regime effects.
Assigning regimes by feel
Seat-of-the-pants classifications prevent reproducibility and allow the analyst to move observations between states after seeing results.
Overfitting recent history
A short sample may contain only one dominant environment and make a fragile relationship appear universal.
Is it for you?
Best for
It is best for quantitative researchers and systematic investors validating indicators across bull and bear markets.
Not ideal for
It is not ideal when historical data are too sparse or inconsistent to define and test regimes reliably.
From the transcript
“You have to quantify it. You can't do it by the seat of your pants.”
“some indicators work very very well in a bear state. Other indicators work very very well in a bull state, but they don't work well…”
“this works or this doesn't work, or this works, but only in these conditions”
From the episode
Interview With Jeff deGraaf: Masters in Business (Audio)