Preregister-and-Replicate Research Protocol
Commit to predictions before analysis and require replication afterward
- Difficulty
- Moderate
- Time to result
- ~months to results
- Steps
- 6
- Confidence
- 95%
This protocol separates confirmatory research from after-the-fact storytelling. Before inspecting the results, researchers publicly or privately register their hypotheses, outcome measures, exclusions, and analytical methods. That commitment constrains the freedom to test many alternatives and report only the combinations that appear successful. All preregistered outcomes should then be reported, including null or negative findings, while unexpected discoveries are clearly labeled exploratory. Finally, the result must survive replication before it is treated as reliable knowledge. The mechanism addresses both direct cherry-picking within a study and publication bias across studies: without null results, the visible literature becomes a selected sample of unusually positive findings. Preregistration creates accountability, while replication tests whether the apparent effect persists beyond the original sample.
Origin
Miller proposed preregistration and replication while discussing the reproducibility crisis and the selective publication of positive academic findings. Extracted from Masters in Business.
Core principles
- 01Predictions should be recorded before outcomes are inspected
- 02Null findings contain valuable information
- 03One significant result is not automatically true
- 04Replication limits both conscious and unconscious cherry-picking
How to run it
- 1
Frame the confirmatory question
State the hypothesis, target population, intervention or exposure, and expected outcome before looking at the relevant results.
Pro tip Make the prediction precise enough that success and failure are distinguishable.
Watch out Broad predictions can be reinterpreted to fit almost any outcome.
- 2
Specify the protocol
Record the sample plan, measures, exclusion criteria, transformations, statistical model, and decision threshold.
Pro tip Include how missing data and outliers will be handled.
Watch out Unspecified analytical choices preserve hidden degrees of freedom.
- 3
Register before analysis
Timestamp the protocol in a repository or registry before outcome analysis begins. Preserve the original entry even if the plan later changes.
Pro tip Register code or pseudocode when the analysis is complicated.
Watch out A registration created after results are visible does not constrain hindsight.
- 4
Execute and disclose deviations
Follow the registered process and document every departure with a reason. Keep confirmatory results separate from exploratory discoveries.
Pro tip Exploratory findings can become preregistered hypotheses for the next study.
Watch out Presenting an unexpected pattern as an advance prediction recreates the original problem.
- 5
Report the complete result set
Publish or archive positive, negative, and ambiguous outcomes from the registered plan.
Pro tip Make data and code available when privacy and legal constraints permit.
Watch out Suppressing null results creates publication and survivorship bias.
- 6
Require replication
Test the finding again with fresh data, preferably through an independent team or setting. Update confidence based on the combined evidence.
Pro tip Use a replication design that preserves the original test while probing generalizability.
Watch out Preregistration reduces bias but cannot guarantee that one sample reflects a true effect.
In the wild
A product team plans to test a redesigned checkout and could measure conversion, basket size, abandonment, support contacts, and return rates across many user groups. It preregisters the primary outcome, sample size, exclusions, and stopping rule. The redesign does not improve the primary outcome, although one unplanned subgroup looks promising; the team reports the null result and treats the subgroup pattern as a hypothesis for a new test.
→ The team avoids shipping a change based on a selectively favorable metric.
A research group finds an attention intervention improves decision quality in one student sample. It archives the preregistered null and secondary results, then asks another laboratory to repeat the primary protocol with a new population. The second study estimates a smaller effect, allowing both teams to report a more realistic combined result.
→ The claimed effect is calibrated using complete and independently reproduced evidence.
Common mistakes
Registering only the headline metric
Leaving exclusions, stopping rules, subgroups, and model choices unspecified still allows analysts to search for a favorable result.
Hiding null findings
Unpublished negative results make the surviving literature appear more successful and reliable than the full research record.
Treating preregistration as proof
A preregistered result can still arise from sampling noise, measurement error, or a flawed design and therefore needs replication.
Is it for you?
Best for
Researchers running experiments or observational studies with multiple outcomes, models, or analytical choices.
Not ideal for
Early exploratory work whose purpose is generating hypotheses rather than testing fixed predictions.
From the transcript
“You register your predictions, and so you know your hands are tied.”
“still valuing even whatever your conclusion is, don't take that as truth when we have to make sure it also replicates”
“It's a huge issue because then you get this kind of implicit cherry picking”
From the episode
Joshua Miller Discusses the Hot Hand Phenomenon (Podcast)