Decision Feedback Audit
Preserve predictions, compare them with outcomes, and improve the decision process
- Difficulty
- Moderate
- Time to result
- ~months to results
- Steps
- 6
- Confidence
- 94%
The Decision Feedback Audit turns experience into deliberate learning. Before acting, preserve the forecast, evidence, assumptions, and selection criteria rather than reconstructing them after the outcome is known. Set a review date and define what success or failure will look like. When results arrive, compare them with the original record, measure calibration, and identify repeated misses such as overconfidence, poor interviews, or neglected information. Aggregate many decisions so vivid anecdotes do not dominate the analysis. The resulting evidence should change the operating process: revise scouting criteria, structure interviews, alter forecast ranges, or introduce blind assessment. The audit then repeats, allowing the organization to test whether the revised method actually improves decisions instead of merely sounding more sophisticated.
Origin
Extracted from Masters in Business. Thaler describes an NFL team that stored years of scouting reports in file cabinets but had never compared those reports with player outcomes.
Core principles
- 01Learning requires feedback
- 02Experience without review does not guarantee improvement
- 03Historical predictions are valuable calibration data
- 04Evaluate the process, not only memorable successes
- 05Blind spots persist when organizations discard evidence
How to run it
- 1
Capture the decision prospectively
Record the prediction, confidence level, evidence, alternatives, and reasons before the outcome is known.
Pro tip Use a standard template so decisions can be compared later.
Watch out Retrospective explanations are vulnerable to hindsight bias.
- 2
Define feedback
Choose measurable outcomes and a review horizon appropriate to the decision.
Pro tip Include both immediate and delayed indicators when performance unfolds over time.
Watch out Changing success criteria after seeing the result destroys the audit.
- 3
Preserve the evidence
Store reports, interview ratings, models, and assumptions in a searchable system linked to the eventual outcome.
Pro tip Capture independent assessments before group discussion.
Watch out A room of unreviewed files is storage, not organizational learning.
- 4
Compare forecast with reality
Review predictions against outcomes across many cases and quantify accuracy, calibration, and recurring failure patterns.
Pro tip Segment results by decision type, evaluator, and confidence level.
Watch out Do not infer process quality from a single lucky success or unlucky failure.
- 5
Redesign the process
Modify criteria, information flow, incentives, or assessment methods to address the observed errors.
Pro tip Test one or two changes at a time where practical.
Watch out Training alone may not overcome biases embedded in the decision environment.
- 6
Audit the revision
Repeat the comparison after enough new decisions accumulate to determine whether the change improved results.
Pro tip Maintain a stable baseline metric across review cycles.
Watch out Do not institutionalize a new method merely because it is more technological.
In the wild
An NFL team retained old scouting reports but had never systematically studied whether its scouts' evaluations predicted player performance. Linking those reports to career outcomes would reveal which traits, evaluators, and confidence levels actually carried signal.
→ The team could replace intuition-driven scouting changes with evidence-based process improvements.
A company in the Philippines replaced first-round human interviews with an AI process designed to extract the relevant signals. Retention increased, indicating that the previous interviews had been missing useful information.
→ The company improved retention by evaluating and changing an established hiring practice.
Common mistakes
Collecting without reviewing
Archived decisions produce no feedback unless they are linked to outcomes and analyzed.
Judging isolated outcomes
A good result can follow a poor decision and a bad result can follow a sound one, so patterns matter more than anecdotes.
Rewriting the original rationale
Reconstructing a forecast after the result is known hides miscalibration and hindsight bias.
Is it for you?
Best for
It is best for recurring decisions whose predictions and eventual outcomes can be recorded and compared.
Not ideal for
It is less useful for genuinely one-off decisions with no observable outcome or comparable future application.
From the transcript
“the only way you can learn anything is feedback, and most of us don't bother, so we don't get the feedback.”
“Oh, old scouting reports. Our eyes are getting back, you know, like, well, have you ever studied those? No?”
“do they invest a little research in improving the process of picking players? No”
From the episode
Special Edition: Nobel Prize Winner Richard Thaler Live from the Economic Club of New York