Early-Success Calibration Loop
Separate early luck from skill before confidence outruns evidence
- Difficulty
- Moderate
- Time to result
- ~weeks to results
- Steps
- 5
- Confidence
- 96%
This model explains why confidence can surge shortly after someone begins learning. A novice initially knows that the task is unfamiliar, but a few early successes create an appealing theory of how it works. Because the evidence is sparse and luck is easy to overlook, confidence rises faster than competence. Continued practice then exposes exceptions and failures, forcing the learner to abandon the premature theory and build a more accurate one. The practical response is to treat early success as provisional, gather a larger sample of outcomes, and compare confidence with demonstrated accuracy. Calibration improves when learners actively distinguish luck from repeatable skill and continue testing their assumptions against feedback.
Origin
David Dunning described research with Carmen Sanchez that tested the popular “Mount Stupid” learning curve using a novel zombie-diagnosis task. Extracted from Masters in Business.
Core principles
- 01Beginners often start with appropriate uncertainty.
- 02A few successes may reflect luck rather than skill.
- 03Sparse evidence encourages premature theories.
- 04Sustained feedback corrects inflated confidence.
- 05Accurate self-assessment grows with task competence.
How to run it
- 1
Declare Beginner Uncertainty
Acknowledge that the task is new and state what you do not yet understand. Give your current confidence a provisional rating.
Pro tip Write the confidence estimate down before seeing the outcome.
Watch out General experience from another field may not transfer cleanly.
- 2
Collect Multiple Outcomes
Perform enough trials to reveal variation rather than judging yourself from one or two successes. Record both results and the conditions surrounding them.
Pro tip Use the same metric across attempts so improvement is comparable.
Watch out A short winning streak can still be random.
- 3
Separate Luck from Skill
Ask whether the success is repeatable and whether your method explains failures as well as wins. Look for outcomes that would disprove your current theory.
Pro tip Repeat the task under slightly different conditions.
Watch out Do not credit every success to skill and every failure to bad luck.
- 4
Stress-Test the Theory
Use new cases and feedback to test the explanation you formed. Revise or discard it when it fails to predict results.
Pro tip Seek cases where an experienced practitioner expects novices to fail.
Watch out A coherent explanation can feel correct even when it is wrong.
- 5
Calibrate Confidence
Compare predicted and actual performance over time. Raise confidence only when accuracy and repeatability improve together.
Pro tip Maintain a prediction log to expose persistent overconfidence.
Watch out Learning can stall when experience stops producing new information.
In the wild
Participants entered a simulated post-apocalyptic world and diagnosed who had a zombie disease. Early lucky successes encouraged them to form a theory from sparse evidence and become overconfident. More trials exposed the theory's weaknesses and corrected their self-assessment.
→ Confidence initially rose too quickly, then declined toward a more realistic level as experience accumulated.
A pilot with several hundred flight hours has enough successful experience to feel capable but may not yet recognize the full range of hazards. Treating that confidence as provisional encourages more conservative choices and continued supervision.
→ The pilot avoids entering high-risk conditions before competence is mature.
Common mistakes
Generalizing from a Lucky Streak
A few favorable outcomes are treated as proof that the method has been mastered, even though chance may explain them.
Protecting the First Theory
The learner explains away contradictory evidence instead of revising the premature model.
Tracking Confidence Without Accuracy
Feeling more familiar with the task is mistaken for becoming reliably better at it.
Is it for you?
Best for
It is best for people learning a new skill or making repeated judgments with observable feedback.
Not ideal for
It is not ideal for one-off decisions where outcomes are delayed or reliable feedback is unavailable.
From the transcript
“But the problem is that you have a few successes they're probably due to luck more than skill, and you think you have it.”
“That is, people arrive at a theory based on data which is far too early, far too sparse, and far too unreliable, but they think…”
“And then experience basically is correcting your flattering impression of your skill as time goes on”
From the episode
David Dunning on the Dunning-Kruger Effect
David Dunning