Repeated-Decision Skill Test
Separate skill from luck by examining repeated decisions and fast feedback
- Difficulty
- Moderate
- Time to result
- ~ongoing to results
- Steps
- 6
- Confidence
- 96%
A favorable result does not reveal whether the underlying decision was skilled or lucky, especially when the outcome takes years to emerge. The Repeated-Decision Skill Test shifts attention from headline performance to the number, comparability, and cadence of decisions. Evaluators identify a recurring choice, reconstruct the process used before outcomes were known, and examine how frequently the person received actionable feedback. Repeated trials make it easier to detect consistency, while shorter feedback loops allow the decision-maker to refine methods deliberately. The framework also asks whether the process changes appropriately over time rather than merely counting wins and losses. Confidence in skill should rise when comparable decisions repeatedly follow a sound process and improve with feedback; it should remain low when a track record rests on a few slow, path-dependent outcomes.
Origin
Extracted from Masters in Business, where David Snyderman reflected on what he wished he had understood earlier about luck, skill, feedback, and investment track records.
Core principles
- 01Outcomes alone do not prove decision quality
- 02Long feedback cycles make luck difficult to separate from skill
- 03Repeated comparable decisions provide stronger evidence
- 04Faster feedback accelerates deliberate improvement
- 05Evaluate the process and how it changes over time
How to run it
- 1
Define the decision class
Specify the recurring judgment being assessed, including its inputs, constraints, and intended result. Exclude outcomes produced by materially different decisions.
Pro tip Use a narrow definition first, then broaden only when the processes are genuinely comparable.
Watch out Grouping unrelated choices creates a misleading sample.
- 2
Reconstruct the process
Record what information was available and how the choice was made before the outcome became known. Separate the decision rule from the eventual result.
Pro tip Use contemporaneous investment memos or decision journals when available.
Watch out Hindsight can make an improvised choice appear systematic.
- 3
Measure repetition
Count how often the person made the same type of decision under comparable conditions. More independent trials provide stronger evidence of repeatability.
Pro tip Note whether apparent repetitions share one hidden macro bet.
Watch out Many positions can still represent a single underlying decision.
- 4
Measure feedback speed
Determine how quickly each choice generated reliable evidence. Favor environments where mistakes become visible soon enough to influence the next attempt.
Pro tip Create interim indicators when final outcomes naturally take years.
Watch out Fast but noisy feedback can teach the wrong lesson.
- 5
Look for adaptation
Compare the process over time to see whether feedback produced sensible refinements. Distinguish disciplined learning from reactive strategy changes.
Pro tip Require an explicit explanation for each meaningful process change.
Watch out Changing methods after every loss can destroy a valid edge.
- 6
Calibrate confidence
Judge skill using process quality, repeated outcomes, independence, and learning speed together. Keep confidence modest when evidence is sparse.
Pro tip Express conclusions probabilistically rather than labeling someone simply skilled or lucky.
Watch out A prestigious or profitable track record can create excessive confidence.
In the wild
One manager produced a large gain from a single ten-year investment, while another made dozens of comparable underwriting decisions, documented each process, received timely repayment data, and improved loss estimates over successive vintages. The first outcome may be excellent, but the second record provides clearer evidence about repeatable skill.
→ The allocator assigns greater confidence to the manager with repeated decisions and observable learning.
Common mistakes
Equating outcome with process
A profitable outcome can follow a poor decision, while a good probabilistic decision can still lose.
Counting correlated bets separately
Several positions driven by one macro event do not provide independent evidence of repeated skill.
Ignoring feedback latency
When results take a decade, the decision-maker has few opportunities to learn whether the process works.
Is it for you?
Best for
It is best for evaluating investors, operators, and professionals whose records contain repeated comparable decisions.
Not ideal for
It is not ideal when decisions are unique, outcomes remain unresolved, or the historical process cannot be reconstructed.
From the transcript
“I think about luck versus skill a lot more than I ever did before.”
“If you're able to have a much faster feedback loop now, you can really hone your skills and understand whether you're making good decisions or…”
“we really try to think about how often do they get to make the same decision and what's the process around that decision and how…”
From the episode
David Snyderman on Specialty Finance and Data in Investing
David Snyderman