Population-Before-Anecdote Puzzle Method
Test competing explanations before generalizing from a compelling story
- Difficulty
- Moderate
- Time to result
- ~weeks to results
- Steps
- 6
- Confidence
- 96%
Damodaran's treatment of buybacks demonstrates a reusable way to investigate corporate puzzles. Start with an observable population-level change rather than an isolated controversy. Generate plausible explanations, such as tax treatment or management compensation, and ask what evidence each explanation predicts. Compare those predictions with historical changes and differences across companies. If buybacks continue after the alleged incentive weakens, or if companies with stronger incentives do not buy back more stock, reject that explanation. Then step back and search for a mechanism that fits the full pattern, including exceptions and changing conditions. This approach does not eliminate judgment, but it prevents a striking anecdote from being mistaken for representative evidence.
Origin
Extracted from Masters in Business through Damodaran's investigation of why corporate cash returns shifted from dividends toward buybacks.
Core principles
- 01A vivid anecdote may not represent the wider population.
- 02A proposed explanation should generate testable predictions.
- 03Historical trends help distinguish causes from coincidences.
- 04Rejected hypotheses are useful progress.
- 05The goal is a general mechanism, not a convenient story.
How to run it
- 1
Define the puzzle
State exactly what changed, over what period, and across which population. Quantify the pattern when possible.
Pro tip Separate the broad trend from the latest newsworthy example.
Watch out A single company or event is not yet a population-level puzzle.
- 2
Generate explanations
List the most credible causal stories without committing to one prematurely.
Pro tip Include the popular explanation as well as less fashionable alternatives.
Watch out Do not confuse an explanation's familiarity with evidentiary support.
- 3
Derive predictions
Specify what should be observable if each explanation is correct, including differences across time or entities.
Pro tip Look for predictions that distinguish competing explanations.
Watch out An unfalsifiable story cannot be meaningfully tested.
- 4
Test against the population
Compare predictions with historical data, cross-sectional evidence, and relevant policy changes.
Pro tip Search for periods when the proposed cause changed but the outcome did not.
Watch out Do not cherry-pick the subset that makes the preferred theory work.
- 5
Reject and refine
Discard explanations that fail and refine the mechanism that best accounts for the broad evidence.
Pro tip Treat a failed hypothesis as information rather than embarrassment.
Watch out Do not retain multiple incompatible causes merely to avoid choosing.
- 6
Generalize cautiously
State the resulting rule, its scope, and its exceptions. Keep the conclusion proportional to the evidence.
Pro tip Name the conditions under which the mechanism should stop working.
Watch out Avoid claiming universality from evidence drawn from one market or era.
In the wild
The hypothesis says managers repurchase shares to improve the value of their stock options. It predicts more buybacks where options represent a larger share of compensation and fewer where options are unimportant. Damodaran reports testing that relationship and finding that the argument did not fit the companies conducting buybacks. The later shift from options to restricted stock also failed to produce the predicted decline in repurchases.
→ The compensation hypothesis was rejected as the general explanation for the buyback trend.
A software company blames one prominent outage for rising churn. An analyst compares churn before and after the outage, across affected and unaffected customers, and against changes in pricing and product usage. The evidence shows that churn began earlier and is concentrated among customers facing a new price tier.
→ The company addresses the broader pricing mechanism instead of treating the outage as the sole cause.
Common mistakes
Starting with the headline anecdote
A dramatic case can anchor the entire investigation even when it is not representative of the population.
Failing to state a prediction
Without an observable implication, an explanation remains a story that can accommodate any result.
Ignoring failed causal timing
If the proposed cause weakens while the outcome persists, the theory needs to be rejected or substantially revised.
Is it for you?
Best for
Analysts investigating persistent changes in markets, organizations, or customer behavior.
Not ideal for
Urgent decisions where no relevant evidence can be gathered and immediate action is unavoidable.
From the transcript
“I said, okay, that's testable. If that is true, buy backs should be greater at companies where management stock options that are a higher percentage…”
“I want to generalize the discussion because it's easy to get trapped in an anecdotal story and draw conclusions that don't apply to the population.”
From the episode
Aswath Damodaran on the Future of Business Education
Aswath Damodaran