Feedback Quality Test
Judge whether practice can produce skill before trusting experience
- Difficulty
- Easy
- Time to result
- ~weeks to results
- Steps
- 6
- Confidence
- 94%
The Feedback Quality Test asks whether an environment can actually teach someone to improve. Practice works well when each attempt produces feedback that is prompt, unambiguous, and consequential, as in a professional tennis match. Many financial decisions fail this test: prices move continuously, short-term outcomes reverse, and a good process can temporarily lose money. Institutional incentives may further distort learning by rewarding agreement with colleagues rather than accurate independent judgment. The framework therefore evaluates the learning environment before treating tenure, confidence, or past returns as evidence of skill. Where feedback is weak, practitioners need longer evaluation periods, written forecasts, process measures, comparison groups, and explicit separation of decision quality from outcome luck.
Origin
Extracted from Masters in Business, using Jason Zweig's comparison between professional tennis feedback and the noisy feedback produced by financial markets.
Core principles
- 01Experience creates skill only when feedback is informative
- 02Useful feedback is prompt, unambiguous, and consequential
- 03Noisy outcomes can reward bad decisions and punish good ones
- 04Performance must be evaluated over a suitable time horizon
- 05Career incentives can replace truth with the wrong feedback signal
How to run it
- 1
Define the decision
State exactly what action, forecast, or skill is being evaluated and over what horizon.
Pro tip Record the reasoning before the outcome becomes known.
Watch out Vague predictions can always be reinterpreted after the fact.
- 2
Test promptness
Determine how quickly the environment reveals whether the decision was effective.
Pro tip Match the review date to the mechanism that should produce the result.
Watch out An immediate price movement may be irrelevant to a long-term investment thesis.
- 3
Test clarity
Ask whether the observed result has one clear interpretation or could have arisen from noise, luck, or unrelated factors.
Pro tip Use multiple observations and comparison benchmarks where possible.
Watch out A profitable outcome does not automatically prove that the original reasoning was sound.
- 4
Test consequence
Identify whether errors matter enough to prompt attention, adaptation, and changed behavior.
Pro tip Use bounded stakes or simulations when real errors would be catastrophic.
Watch out Feedback that is painless or ignored may not create learning.
- 5
Audit incentives
Check whether the practitioner is rewarded for accuracy, customer outcomes, conformity, or career protection.
Pro tip Track absolute decision quality separately from performance relative to peers.
Watch out Career-safe decisions can be personally rational while remaining suboptimal for clients.
- 6
Adjust the learning system
When natural feedback is weak, add decision journals, delayed reviews, base-rate comparisons, and process metrics.
Pro tip Grade both the decision process and the eventual outcome.
Watch out Do not manufacture certainty where the environment remains fundamentally unpredictable.
In the wild
A firm reviews a manager after one successful quarter. Applying the test reveals that the horizon is too short, market movements are noisy, and the manager took the same crowded positions as peers. The firm instead reviews written theses over several market regimes and distinguishes process quality from luck.
→ The firm avoids labeling short-term outperformance as proven investment skill.
A sales team tests two scripts across comparable lead groups. Responses arrive quickly, conversions are objectively recorded, and representatives receive coaching after each batch.
→ The clear feedback loop allows deliberate practice to improve performance within weeks.
Common mistakes
Equating outcome with process
A good result may follow a poor decision, especially in noisy environments. Evaluate what was knowable when the choice was made.
Using the wrong time horizon
Feedback arriving too early or too late can conceal the relationship between action and result.
Ignoring incentive distortion
People learn to optimize whatever their institution rewards, which may be conformity rather than accuracy.
Is it for you?
Best for
It is best for evaluating forecasting, investing, management, and other activities where outcomes may be delayed or noisy.
Not ideal for
It is unnecessary for simple tasks that provide immediate, objective, and repeatable feedback.
From the transcript
“The feedback is noise.”
“the second essential criterion, the other thing you have to have, is good feedback.”
“The consequences for you aren't in being right or wrong. It's whether you're right or wrong relative to the crowd.”
From the episode
An Interview With Jason Zweig: Masters in Business (Audio)
Jason Zweig