Simplicity Before Multiple Comparisons
Control hypothesis count before fitting scarce observations
- Difficulty
- Moderate
- Time to result
- ~weeks to results
- Steps
- 6
- Confidence
- 92%
Simplicity Before Multiple Comparisons is a safeguard against overfitting. Before accepting a predictive pattern, the researcher compares the amount of evidence with the number of variables, equations, transformations, and hypotheses explored. Every additional comparison creates another opportunity to discover an impressive result by chance. The remedy is to predefine the main hypothesis, reduce unnecessary parameters, preserve unseen validation data, and favor the simplest model that explains the observations adequately. Complexity is admitted only when it produces repeatable improvement outside the fitting sample. The output is a smaller, more falsifiable model whose performance is less dependent on accidental features of one historical period.
Origin
Extracted from Masters in Business, where Victor Niederhoffer compared overcomplicated trading systems with epicyclic astronomical models containing too many equations.
Core principles
- 01Prefer the simplest adequate explanation
- 02Match model complexity to available evidence
- 03Count every hypothesis tested
- 04Demand validation outside the fitting sample
How to run it
- 1
Declare the Hypothesis
Write down the primary relationship you expect before exploring the data. Define the variables, direction, horizon, and success criterion.
Pro tip Register one primary test and label all later discoveries exploratory.
Watch out Changing the hypothesis after seeing results hides the true number of comparisons.
- 2
Inventory the Evidence
Count independent observations and assess their coverage across relevant regimes. Distinguish genuinely independent evidence from repeated measurements of the same event.
Pro tip Use effective sample size rather than the largest available row count.
Watch out A large dataset can still contain very few independent market episodes.
- 3
Count the Complexity
List every parameter, transformation, filter, subgroup, and hypothesis considered. Treat abandoned trials as part of the research process, not as nonexistent tests.
Pro tip Maintain a research log of every variation attempted.
Watch out Unrecorded experimentation makes chance discoveries look intentional.
- 4
Simplify the Model
Remove variables and rules that do not provide stable incremental value. Prefer the smallest specification that retains the essential mechanism.
Pro tip Compare each added component against a simple baseline.
Watch out Do not preserve complexity merely because it improves in-sample fit.
- 5
Validate on Untouched Data
Test the simplified model on observations not used for discovery or tuning. Require useful performance after realistic costs and reasonable parameter changes.
Pro tip Use rolling or forward validation when temporal order matters.
Watch out Repeatedly checking the same holdout set eventually turns it into training data.
- 6
Reject Fragile Results
Discard patterns that vanish outside the original sample, depend on one parameter value, or lack enough observations for their complexity. Record the rejection to avoid rediscovering it later.
Pro tip Treat a clean rejection as a successful research outcome.
Watch out Do not rescue a failed model by adding another layer of conditions.
In the wild
An amateur tests dozens of indicators, entry times, filters, and holding periods on a small sample. One elaborate combination looks exceptional, but its success reflects the search process rather than a durable effect. Reducing the model to one predeclared signal and testing it on untouched data reveals that the apparent edge does not persist.
→ A fragile in-sample discovery is rejected before capital is exposed.
A researcher hypothesizes that unusually large one-day declines are followed by short rebounds. Instead of optimizing many thresholds, she selects one economically justified definition, keeps recent years untouched, and compares the rule with a simple unconditional baseline.
→ The resulting estimate is easier to interpret and less exposed to multiple-comparison bias.
Common mistakes
Fitting Too Many Variables
Adding parameters to a small sample can explain historical noise perfectly while producing no predictive value.
Ignoring Abandoned Tests
Every attempted hypothesis affects the probability of finding a chance result. Reporting only the winner exaggerates confidence.
Optimizing the Holdout Set
Repeatedly revising a model after viewing validation results contaminates the supposed out-of-sample test.
Is it for you?
Best for
It is best for analysts, traders, and data scientists developing predictive models from limited historical data.
Not ideal for
It is not intended to prohibit justified complexity when abundant evidence and robust validation support it.
From the transcript
“Too too much complexity with simplicity will do”
“They call it multiple comparisons.”
“then they try to fit that to a very small,”
From the episode
Victor Niederhoffer: Lessons of Making and Losing a Fortune
Victor Niederhoffer