Noise Audit
Measure hidden disagreement before redesigning organizational judgment
- Difficulty
- Easy
- Time to result
- ~days to results
- Steps
- 6
- Confidence
- 98%
A Noise Audit measures how differently professionals judge the same realistic cases. Leaders first estimate how much disagreement they expect, establishing a benchmark for tolerable variation. A representative group then evaluates identical cases independently, preventing discussion, hierarchy, and anchoring from manufacturing superficial consensus. The organization compares the resulting spread with its expectation and examines where judgments diverge most. This converts an abstract concern about decision quality into measurable evidence. The audit does not determine which individual answer is correct, nor does it directly measure shared bias. Instead, it diagnoses inconsistency within a judgment system and creates a baseline against which structured procedures, averaging, rules, feedback, or other interventions can be tested.
Origin
Kahneman conceived the method while consulting for a large insurance company, where underwriters evaluating identical cases differed by 65% rather than the 10% executives expected.
Core principles
- 01Disagreement must be measured before it can be managed.
- 02Experts should judge identical cases independently.
- 03Expected disagreement is often far lower than actual disagreement.
- 04Poor feedback allows noise to remain hidden.
- 05System-level procedures improve judgment more reliably than individual insight.
How to run it
- 1
Choose the Judgment System
Identify a recurring consequential judgment made by several professionals, such as pricing risk, assessing claims, or evaluating candidates.
Pro tip Prioritize decisions with comparable inputs and material financial or human consequences.
Watch out Do not audit tasks where a calculation already supplies an objectively correct answer.
- 2
Build Realistic Cases
Have domain experts construct several representative cases containing the information normally available to decision-makers.
Pro tip Use cases credible enough that participants treat them like real work.
Watch out Artificial or incomplete cases may create variation unrelated to the live process.
- 3
Record Expected Variation
Ask leaders how far apart two randomly selected judgments should reasonably be. Define the comparison metric before seeing results.
Pro tip For quantitative judgments, express the difference as a percentage of the pair's average.
Watch out Do not let participants revise expectations after observing the measured spread.
- 4
Collect Independent Judgments
Give the same cases to multiple professionals and collect their answers separately. Prevent participants from seeing or discussing other answers first.
Pro tip Use enough judges and cases to expose persistent variation rather than one unusual disagreement.
Watch out Sequential discussion creates correlated answers and conceals the system's true noise.
- 5
Measure the Dispersion
Calculate how much judgments differ for each case and across the system. Compare actual variation with the expected level.
Pro tip Report both an intuitive summary and the full distribution so extreme disagreements remain visible.
Watch out A correct average does not mean the underlying judgments were consistently good.
- 6
Redesign and Recheck
Apply suitable controls such as structured assessments, independent aggregation, rules, or feedback, then repeat the audit.
Pro tip Retain the initial result as a baseline for measuring improvement.
Watch out Reducing variation alone does not eliminate a bias shared by all judges.
In the wild
Executives expected two randomly selected underwriters pricing the same realistic risk to differ by about 10%. Approximately 50 underwriters assessed identical cases independently. Their judgments differed by 65%, more than five times the expected amount, exposing a potentially enormous source of pricing error.
→ The audit revealed that professional consistency was dramatically worse than management assumed and initiated broader research into judgment noise.
A company gives the same anonymized employee records to several promotion-panel members. Beforeward, executives predict modest rating differences. Independent scores reveal major disagreement on leadership potential, prompting the company to define assessment dimensions and gather ratings before panel discussion.
→ The company establishes a measurable baseline and reduces variation in later promotion cycles.
Common mistakes
Letting Judges Confer First
Discussion causes participants to influence one another, concealing true disagreement and reducing the amount of independent information collected.
Treating the Average as Proof of Accuracy
Overestimates and underestimates can produce an apparently correct average while still causing separate harmful decisions.
Confusing Noise with Bias
The audit detects unwanted variability, but a consistent shared error can remain even after noise is reduced.
Is it for you?
Best for
It is best for organizations where multiple people make recurring judgments from similar information without reliable outcome feedback.
Not ideal for
It is not ideal for deterministic tasks governed by calculations or for unique decisions that cannot be compared meaningfully.
From the transcript
“conduct what we call the noise audit, give them the same problem and look at their solution.”
“We predict that you'll find more noise then than you think you will.”
“the correct answer among underwriters in that company with sixty five percent more than five times as much as expected.”
From the episode
Daniel Kahneman on Behavioral Economics (Podcast)
Daniel Kahneman