Policy Evidence Triangulation
Compare multiple studies before converting research into policy advice
- Difficulty
- Moderate
- Time to result
- ~weeks to results
- Steps
- 6
- Confidence
- 91%
Policy Evidence Triangulation treats a research paper as one input rather than a final answer. The analyst begins with a defined policy question, gathers several relevant studies, and explicitly maps where their data, methods, and conclusions agree or conflict. Novel or surprising findings receive additional scrutiny because academic incentives can reward results that stand apart from the literature. Replication evidence, data quality, institutional context, and uncertainty are then incorporated before advice reaches decision-makers. The output is not a pile of citations but a structured account of what is robust, what remains contested, and how much confidence a policy recommendation deserves. Sahm describes this comparative discipline as a core skill learned while preparing analysis for the Federal Reserve boardroom.
Origin
Extracted from Masters in Business. Sahm contrasted the Federal Reserve’s comparative evidence standards with academic incentives for novel findings and the economics profession’s weak replication culture.
Core principles
- 01Never base consequential advice on one paper
- 02Compare agreement as well as disagreement
- 03Treat novel findings as claims requiring extra scrutiny
- 04Distinguish academic incentives from policy needs
- 05Check whether central results can be reproduced
How to run it
- 1
Frame the decision
Translate the policy request into a precise question, affected population, intended outcome, and relevant time horizon.
Pro tip State what evidence would materially change the recommendation.
Watch out A vague question encourages selective use of whatever research is easiest to find.
- 2
Build a comparison set
Collect the focal study plus several credible papers addressing the same mechanism or outcome.
Pro tip Include studies with different datasets, periods, or methods when possible.
Watch out Do not choose comparison papers only because they support the preferred conclusion.
- 3
Map convergence
List the findings, assumptions, and mechanisms on which the studies agree.
Pro tip Separate agreement on direction from agreement on effect size.
Watch out Several papers may share the same flawed dataset or identifying assumption.
- 4
Map disagreement
Identify conflicting results and determine whether they arise from populations, methods, definitions, or genuine uncertainty.
Pro tip Explain disagreement rather than averaging incompatible estimates mechanically.
Watch out Suppressing contrary evidence creates false confidence.
- 5
Test robustness
Review replication attempts, data accessibility, sensitivity checks, and whether the main empirical result can be reproduced.
Pro tip Apply extra scrutiny to findings that are both surprising and policy consequential.
Watch out Publication in a leading journal does not guarantee reproducibility.
- 6
Brief with calibrated confidence
Present the recommendation alongside the strongest supporting evidence, important counterevidence, and unresolved uncertainty.
Pro tip State the conditions under which the recommendation should be revisited.
Watch out Do not convert a tentative result into categorical advice.
In the wild
An analyst reviews a new study finding that enhanced benefits did not slow job returns. Instead of relying on that result alone, the analyst compares other current studies, evidence from earlier recessions, labor-demand conditions, and findings about job matching for disadvantaged groups.
→ The policy brief distinguishes immediate crisis effects from incentive concerns that may emerge later in the recovery.
A paper reports an unusually large behavioral response to a tax change. The policy team compares it with three related studies, checks whether the result has been replicated, and identifies that the estimate depends on a narrow population and unusual period.
→ Decision-makers receive a bounded recommendation instead of treating the headline estimate as universal.
Common mistakes
Taking one paper into the boardroom
A single study may be unrepresentative, fragile, or contradicted by the broader evidence base.
Counting citations instead of comparing them
A bibliography is not synthesis unless the analyst explains where studies converge and why they differ.
Trusting novelty without replication
Surprising findings deserve more verification, not less, especially when they could drive consequential decisions.
Is it for you?
Best for
It is best for analysts translating academic literature into institutional or government decisions.
Not ideal for
It is not ideal when an emergency requires immediate action and no meaningful comparison evidence yet exists.
From the transcript
“here's my paper, here's three other papers, here's how they agree, here's how they disagree”
“I never took a result, a piece of advice into the boardroom that was one paper”
“Academia rewards novel surprising findings. Those are the ones that we absolutely should be checking the map”
From the episode
Claudia Sahm on Economic Policy (Podcast)
Claudia Sahm