Anonymous Iterative Team Forecasting
Preserve independent judgment while using structured rounds of team criticism
- Difficulty
- Moderate
- Time to result
- ~weeks to results
- Steps
- 6
- Confidence
- 96%
Anonymous iterative team forecasting combines independent estimates with structured interaction. Each participant first submits a probability and rationale without seeing colleagues' answers. The submissions are anonymized and circulated, preventing senior or charismatic members from dominating the discussion. Participants inspect the distribution, challenge assumptions through precise questions, and revise their estimates over two or three rounds. A final aggregate captures both the diversity of initial judgments and the improvements generated through criticism. The approach resembles an enhanced Delphi process: anonymity protects independence, while iteration allows useful information to spread. Its value should be tested against a simple average of the initial estimates and verified through subsequent scoring rather than assumed from team satisfaction.
Origin
Extracted from Masters in Business, where Philip Tetlock explained why structured teams in forecasting tournaments outperformed independent forecasters and prediction markets.
Core principles
- 01Collect independent judgments before discussion
- 02Separate ideas from the status of their authors
- 03Use disagreement to uncover hidden assumptions
- 04Iterate after participants inspect competing rationales
- 05Compare the structured result with a simple crowd average
How to run it
- 1
Frame the decision
Define a resolvable question, deadline, and shared evidence cutoff. Ensure every participant is forecasting the same event.
Pro tip Publish the resolution criteria before collecting judgments.
Watch out Small wording differences can cause participants to answer different questions.
- 2
Collect independent estimates
Have each person privately submit a probability and concise rationale before any group discussion. Preserve the initial judgments for later comparison.
Pro tip Ask participants to identify their strongest assumption and biggest uncertainty.
Watch out A preliminary meeting can anchor the supposedly independent estimates.
- 3
Anonymize and circulate
Remove names and status indicators, then distribute all estimates and explanations to the group. Display the range as well as the average.
Pro tip Randomize the order of submissions in each round.
Watch out Writing style or role-specific details may accidentally reveal authorship.
- 4
Question assumptions
Invite participants to identify disagreements and ask targeted questions about evidence, base rates, and causal mechanisms. Keep criticism focused on claims rather than people.
Pro tip Prioritize assumptions that explain the widest probability differences.
Watch out Anonymity should not become permission for dismissive or hostile criticism.
- 5
Revise through rounds
Collect revised probabilities and rationales after participants review the feedback. Repeat for two or three rounds, stopping when new information no longer changes judgments materially.
Pro tip Require a short note explaining every substantial revision.
Watch out Do not pressure the group to converge merely for appearance's sake.
- 6
Aggregate and evaluate
Combine the final estimates using a predefined method and compare the result with the unweighted average of the first round. Score both after resolution to learn whether iteration added value.
Pro tip Retain dissenting estimates because they may contain information suppressed by aggregation.
Watch out A smoother consensus is not evidence of a more accurate forecast.
In the wild
Executives independently estimate the probability that a major supplier will miss a delivery deadline and provide evidence. Their anonymous rationales reveal that operations knows about a capacity constraint while finance knows the supplier secured emergency funding. After two rounds of questioning and revision, the estimates are aggregated.
→ The company obtains a better-informed probability without allowing the most senior executive's initial view to anchor the group.
Participants submit and discuss probability judgments in structured teams that encourage open-minded criticism. Their combined work is scored against outcomes and compared with individual forecasts and prediction markets.
→ Structured superforecaster teams outperform less selective or less collaborative forecasting methods.
Common mistakes
Discussing before independent estimates
Early conversation allows the first speaker or highest-status participant to anchor everyone else.
Forcing consensus
Convergence can erase valid minority information and create a false appearance of certainty.
Leaving the team unscored
Without outcome-based comparison, the group cannot know whether interaction improved on the initial crowd average.
Is it for you?
Best for
It is best for teams whose members hold distributed knowledge but differ in status, confidence, or communication style.
Not ideal for
It is not ideal for emergencies requiring an immediate command decision or questions without independently held information.
From the transcript
“But there's a way to get the benefits of independence and the benefits of creative interaction at the same time.”
“And one way to do that is by going getting everybody to make their judgments anonymously.”
“So the high status guy isn't swaying everything the way off. It often happens in groups.”
From the episode
Barry Ritholtz's Masters in Business: Philip Tetlock Interview
Philip Tetlock