Hypothesis-First Signal Research
Explain the inefficiency before optimizing the data
- Difficulty
- Advanced
- Time to result
- ~months to results
- Steps
- 6
- Confidence
- 97%
This process starts with an economic story: why should a particular characteristic predict returns, and what market inefficiency or behavioral error produces the effect? Researchers then examine data to determine how best to measure and capture that mechanism. Multiple formulations can be tested, including machine-learning interpretations of richer datasets, but model flexibility raises the danger of overfitting. The research process therefore needs statistical and procedural safeguards, out-of-sample validation, and comparison against simpler baselines. A model should survive because its mechanism and evidence generalize, not because it perfectly explains one historical window. The final output is a documented signal with a rationale, measurement rule, validation record, implementation context, and explicit failure conditions.
Origin
Extracted from Masters in Business, where John Chisholm explained Acadian's sequence for developing investment signals and controlling overfitting.
Core principles
- 01Every signal needs a plausible mechanism
- 02Data should test an idea rather than manufacture one
- 03Measurement can evolve while the underlying hypothesis remains stable
- 04Historical fit is not evidence of future robustness
- 05Research safeguards must match model flexibility
How to run it
- 1
State the hypothesis
Describe why a characteristic should be related to future returns. Name the investor behavior, information gap, or market friction expected to create the opportunity.
Pro tip Write the hypothesis before inspecting detailed results.
Watch out A vague story added after a successful backtest does not provide meaningful support.
- 2
Define the observable evidence
Choose data that can measure the proposed mechanism across a relevant universe and time period. Establish data-quality and availability requirements.
Pro tip Confirm that the data would genuinely have been available at each historical decision point.
Watch out Look-ahead bias can make an unusable signal appear powerful.
- 3
Test multiple measurements
Compare reasonable ways to express the characteristic, such as different momentum horizons or risk adjustments. Keep each variation tied to the original hypothesis.
Pro tip Include a simple specification as a baseline.
Watch out Testing unlimited variations increases the probability of discovering a spurious winner.
- 4
Validate beyond the fitting sample
Evaluate the model on unseen periods, markets, and security groups. Look for a coherent effect rather than perfect uniformity.
Pro tip Use rolling or genuinely held-out tests to simulate future deployment.
Watch out The best historical fit may be the least robust live model.
- 5
Apply complexity safeguards
Use statistical and research procedures that penalize unnecessary flexibility, especially for machine-learning models. Compare their performance with conventional linear alternatives.
Pro tip Choose the simplest model that captures the repeatable effect adequately.
Watch out Neural networks and other flexible models can memorize historical noise.
- 6
Document and monitor failure modes
Specify conditions under which the hypothesis could weaken, become arbitraged, or stop being implementable. Monitor live results against those expectations.
Pro tip Distinguish normal variation from evidence that the mechanism has changed.
Watch out Do not preserve a signal solely because it once had an excellent backtest.
In the wild
A research team hypothesizes that investors adjust slowly to new information. It compares conventional twelve-month risk-adjusted momentum with several price-path features, validates them on unseen markets, and requires a machine-learning model to beat a linear baseline after costs.
→ The team adopts a more robust momentum signal without allowing algorithmic flexibility to replace economic reasoning.
Researchers propose that aggregated parking-lot activity can provide an early view of retailer revenue. They verify location coverage, timestamp integrity, forecast horizon, and out-of-sample performance before assessing whether the result is large enough to affect the total portfolio.
→ The dataset is adopted only if its predictive value and portfolio contribution justify its infrastructure cost.
Common mistakes
Starting with an attractive chart
A pattern without a prior mechanism may be the product of data mining. Begin with a falsifiable reason that the relationship should exist.
Selecting the best in-sample model
Flexible algorithms can produce excellent historical results by fitting noise. Require unseen-data validation and simple-model comparisons.
Ignoring live usability
A predictive relationship may still be unusable because of delayed data, limited coverage, high costs, or insufficient portfolio impact.
Is it for you?
Best for
It is best for quantitative teams evaluating new factors, alternative datasets, or machine-learning models.
Not ideal for
It is not ideal for purely discretionary decisions that cannot be expressed as testable hypotheses.
From the transcript
“The first first step is always is there a you think it as a story, it's really hypothesis of why a particular characteristics related to…”
“And when you do that, you have to be careful because machine learning is one way to do what's called overfitting a problem, you know,…”
“We have various, you know, statistical procedures that we follow, UM in various research procedures we followed to try to avoid that, but it does…”
From the episode
Acadian’s John Chisholm Discusses Investments
John Chisholm