Ground-Truth Simulation Benchmark
Test an analysis against simulated worlds where the truth is known
- Difficulty
- Advanced
- Time to result
- ~weeks to results
- Steps
- 6
- Confidence
- 98%
The method validates an analytical procedure by applying it to synthetic data generated from a world whose underlying mechanism is known. First define the phenomenon the analysis claims to detect, then construct controlled scenarios where that phenomenon is absent, present, or varies in strength. Generate many data sets from each scenario and run the unchanged analysis on them. Because the ground truth is known, discrepancies expose bias, weak statistical power, or misleading benchmarks. Miller and Sanjurjo used coin flips as a controlled analogue for basketball shots: the flips were independent by construction, yet the conventional hot-hand calculation still produced an apparent post-streak decline. That unexpected result revealed that the analytical benchmark—not the arithmetic applied to the observed shots—was wrong.
Origin
Joshua Miller and Adam Sanjurjo developed the approach after the original hot-hand test failed to identify Craig Hodges's visibly unusual shooting streaks. Extracted from Masters in Business.
Core principles
- 01Observed data alone cannot reveal how an estimator behaves
- 02A valid benchmark requires a world with known ground truth
- 03Synthetic data exposes bias hidden by intuitive interpretations
- 04Robust methods should behave sensibly across multiple plausible worlds
How to run it
- 1
Define the target mechanism
State exactly what the analysis is intended to detect and identify the observable statistic used as its proxy.
Pro tip Separate the latent state, such as being hot, from the noisy outcome used to measure it.
Watch out A vague target makes it impossible to judge whether the analysis succeeds.
- 2
Build a null world
Create a model where the target mechanism is known not to exist. Specify the data-generating process rather than merely reshuffling observed data.
Pro tip Start with the simplest possible model, such as independent coin flips.
Watch out Do not assume that an intuitive statistic will be unbiased under the null.
- 3
Generate repeated synthetic data
Simulate many samples with the same relevant size and structure as the real data. Preserve features such as sequence length and stopping rules when they affect selection.
Pro tip Use enough repetitions to distinguish a systematic effect from simulation noise.
Watch out An unrealistic sample structure can conceal the bias being tested.
- 4
Run the unchanged analysis
Apply the exact estimator, conditioning rule, or comparison used on the real observations. Record what it concludes in each known world.
Pro tip Automate the pipeline so the real and simulated data receive identical treatment.
Watch out Changing the analysis after seeing simulated results can introduce a new form of cherry-picking.
- 5
Compare output with truth
Measure the gap between the analytical result and the known data-generating mechanism. Determine whether the procedure is biased, underpowered, or correctly calibrated.
Pro tip Calculate both the expected bias and its variation across samples.
Watch out A correctly calculated statistic can still be interpreted against the wrong benchmark.
- 6
Stress-test alternative worlds
Vary plausible assumptions, mechanism strengths, sample sizes, and noise levels. Retain conclusions only when the method behaves meaningfully across relevant scenarios.
Pro tip Include worlds that contradict your preferred explanation.
Watch out Showing robustness only within one convenient model does not establish general validity.
In the wild
A subscription analyst observes that customers contacted after three active months subsequently cancel at a high rate. Before claiming that outreach causes churn, the analyst simulates customers whose monthly cancellations are independent and applies the same rule of selecting only periods following three active months. The simulation shows how conditioning on a completed streak changes the available observations, providing a benchmark for the real estimate.
→ The analyst distinguishes selection bias from a genuine outreach effect.
Miller and Sanjurjo modeled a shooter as independent coin flips and repeatedly applied the original post-streak shooting-percentage calculation. Although the simulated player had a constant probability of success, the measured percentage was expected to fall after a streak. The unchanged shooting percentage in the original basketball data therefore represented performance above the correct null benchmark.
→ A result previously interpreted as no hot hand became evidence consistent with a substantial hot-hand effect.
Common mistakes
Trusting the intuitive benchmark
Analysts may assume that an independent process must produce an estimated post-streak rate equal to its unconditional rate. Conditioning and finite-sample selection can make that benchmark wrong.
Testing only the observed world
Reanalyzing one real data set in several ways does not show how the procedure behaves when the underlying truth changes.
Confusing outcomes with latent states
A single noisy outcome may be a weak measure of an underlying condition such as confidence, skill, or being in the zone.
Is it for you?
Best for
Researchers and analysts validating statistical tests, estimators, forecasting methods, or pattern-detection rules.
Not ideal for
Questions that cannot be represented with explicit assumptions or measurable simulated outcomes.
From the transcript
“you have to sit down and generate fake data.”
“What if the world looked like this, how would my analysis behave? What if the world look like that? How the analysis behaves?”
“what would this analysis give you if there were no hot hand?”
From the episode
Joshua Miller Discusses the Hot Hand Phenomenon (Podcast)