Question-First Alternative Data Process
Start with an investment controversy, then assemble data that resolves it
- Difficulty
- Moderate
- Time to result
- ~weeks to results
- Steps
- 6
- Confidence
- 93%
The question-first process reverses the common alternative-data workflow. Instead of collecting vast datasets and searching indiscriminately for correlations, the investor begins with a specific controversy whose resolution would affect an investment decision. The team defines what observable evidence would distinguish competing explanations, then finds data capable of measuring that evidence. Each source is examined for context, bias, coverage, and interpretive difficulty before being combined with complementary sources. Computing, storage, and machine learning make the analysis feasible, but they do not determine which question matters. The final test is decision relevance: does the assembled evidence clarify the controversy or alter the investment thesis? This approach reduces data hoarding, limits spurious discovery, and makes it easier to explain why a signal should have economic meaning.
Origin
Extracted from Masters in Business, where Matthew Rothman contrasted data-for-data's-sake with research that begins from a defined investment controversy.
Core principles
- 01Begin with a decision-relevant controversy, not an interesting dataset.
- 02Use data only where it can improve an investment judgment.
- 03Combine complementary sources because no single dataset is a holy grail.
- 04Interpret context and nuance before converting information into a signal.
- 05Treat storage and computing as enablers rather than the research objective.
How to run it
- 1
Frame the controversy
Write the unresolved investment question as competing explanations or outcomes that would lead to different decisions.
Pro tip Make the question narrow enough that evidence could plausibly resolve it.
Watch out A vague desire to find alpha is not a testable controversy.
- 2
Define decisive evidence
Identify the observations that would strengthen, weaken, or falsify each side of the controversy.
Pro tip Specify the expected direction and timing of each observation before examining the data.
Watch out Defining success after seeing the results invites confirmation bias.
- 3
Source relevant data
Search for conventional and unconventional datasets that directly measure the required evidence.
Pro tip Consider transactions, location activity, logistics, imagery, and text when standard reports lag the underlying behavior.
Watch out Novelty does not make a dataset relevant.
- 4
Audit meaning and context
Examine how the data was generated, what population it represents, and where language or domain context can reverse its apparent meaning.
Pro tip Build domain-specific dictionaries and interpretation rules for text.
Watch out The same phrase or measurement may imply different outcomes across markets, industries, or regimes.
- 5
Combine complementary sources
Integrate datasets that illuminate different parts of the causal chain rather than expecting one source to answer everything.
Pro tip Use independent sources to cross-check important claims.
Watch out Correlated sources can create false confidence without adding independent evidence.
- 6
Test decision value
Evaluate whether the combined evidence reliably resolves the controversy and changes portfolio action after realistic costs and delays.
Pro tip Document when the signal should not be used as carefully as when it should.
Watch out A statistically interesting result may still have no practical investment value.
In the wild
An investor questioning whether a retailer's sales weakness is temporary could combine anonymized mall foot traffic, card spending, email receipts, and shipment activity. Rather than buying these datasets because they are novel, the investor selects them because each measures a different link between customer visits, purchases, and inventory movement.
→ The combined evidence can identify persistent deterioration before conventional financial reporting confirms it.
A model evaluating management sentiment encounters the statement that a quarter was not great but exceeded modest expectations. The team supplies domain context, handles the nested qualifications, and compares the language with reported figures instead of assigning a positive or negative score from isolated words.
→ The resulting signal reflects the statement's qualified meaning rather than a misleading keyword count.
Common mistakes
Collecting data without a question
Warehousing every available source creates cost and complexity while encouraging researchers to mistake accidental patterns for useful evidence.
Seeking a single holy-grail dataset
Most investment conclusions require several complementary observations, each with different biases and limitations.
Discarding context
Language, economic regimes, and market conventions can change a measurement's meaning, so mechanically uniform interpretation produces errors.
Is it for you?
Best for
It is best for quantitative investors evaluating satellite imagery, transactions, text, logistics, location, or other unconventional datasets.
Not ideal for
It is not ideal when the research question is undefined or the team lacks the domain knowledge needed to interpret contextual signals.
From the transcript
“But we also need to slow down and start thinking about it from an investor's perspective of what is the data that I need that's…”
“it's really about how you take all these disparate data sets combine them h in a thoughtful manner that's really going to give you your…”
“you have to teach machines context, You have to teach them nuance, you have to teach them the ability to understand when this is bad…”
From the episode
Matthew Rothman
Matthew Rothman