Payoff-Weighted Performance Test
Judge decisions by total gains and losses, not the percentage called correctly
- Difficulty
- Easy
- Time to result
- ~weeks to results
- Steps
- 6
- Confidence
- 93%
The payoff-weighted test evaluates a strategy by combining outcome frequency with outcome magnitude. Instead of asking only how often a prediction was right, record how much was gained on correct calls and how much was lost on incorrect ones. Multiply those magnitudes by their frequencies to estimate total or expected payoff. This catches strategies that generate many small wins but occasionally suffer losses large enough to erase them. It also prevents analysts from flattering performance by grouping decisions into periods where sparse successful calls make the monthly record look strong. Miller illustrates the problem with a rule that predicts a decline after three consecutive gains: it can appear right in many months while losing heavily during the fewer months when the upward run continues.
Origin
Miller described this decision rule while explaining how streak-conditioned predictions can make an unprofitable market strategy appear accurate. Extracted from Masters in Business.
Core principles
- 01Accuracy and profitability are different quantities
- 02A few large losses can erase many small wins
- 03Decision frequency affects aggregate performance
- 04Magnitude must be evaluated alongside direction
How to run it
- 1
Define the decision unit
Specify what counts as one prediction, trade, or operational decision and use that unit consistently.
Pro tip Record every eligible event rather than only periods containing a win.
Watch out Changing the unit of analysis can make the same strategy appear better or worse.
- 2
Capture frequency and magnitude
For each decision, record whether it was correct and the monetary or utility gain or loss it produced.
Pro tip Include fees, slippage, and other implementation costs.
Watch out Directionally correct trades can still lose money after costs.
- 3
Calculate conditional averages
Measure the win rate, average win, loss rate, and average loss. Examine how many decisions occur in each reporting period.
Pro tip Review the distribution as well as the averages to detect tail risk.
Watch out A high win rate is meaningless without the size of losses.
- 4
Compute expected payoff
Combine outcome probabilities and magnitudes to estimate the strategy's payoff per decision and in aggregate.
Pro tip Express the result in both currency and risk-adjusted terms when possible.
Watch out Do not infer positive expectancy from a majority of winning periods.
- 5
Stress-test adverse runs
Model what happens when the strategy's losing condition persists longer than usual. Check whether capital and risk limits can survive those sequences.
Pro tip Use historical extremes and harsher synthetic scenarios.
Watch out Rare losses can dominate lifetime performance.
- 6
Compare with alternatives
Evaluate the strategy against a passive benchmark and simpler rules using the same payoff accounting.
Pro tip Prefer the simpler strategy when payoff differences are not robust.
Watch out Selective time windows can turn random variation into apparent skill.
In the wild
A trader predicts that a stock will decline whenever it has risen for three consecutive sessions. The rule produces correct calls in many months, but those months contain only small gains. During a few persistent rallies, repeated short positions generate much larger losses. A payoff-weighted review counts every prediction and compares the dollars won and lost rather than the number of favorable months.
→ The apparently accurate strategy is correctly identified as unprofitable.
A lead-scoring model correctly rejects many low-value prospects and boasts high classification accuracy, but it occasionally rejects large enterprise accounts. The team weights each outcome by expected contract value rather than treating every classification equally.
→ The company replaces raw accuracy with an economically meaningful performance measure.
Common mistakes
Optimizing the hit rate
Maximizing the percentage of correct calls can encourage many low-value wins while ignoring a few catastrophic losses.
Scoring by winning periods
A strategy can win in most months yet lose money overall when unsuccessful months contain more decisions or larger losses.
Ignoring implementation costs
Fees, spreads, slippage, and capital constraints can turn positive paper expectancy into negative realized performance.
Is it for you?
Best for
Traders, forecasters, fund managers, and operators evaluating repeated decisions with unequal consequences.
Not ideal for
Situations where every correct and incorrect decision has exactly the same payoff and cost.
From the transcript
“You have to pay attention not too how often they're right, but how much money they make when they're right and when they're wrong.”
“But if I were to bet I wouldn't be making any money because I'd be losing a lot of money in the months where I…”
“So it's not your banning average, it's how far the ball goes when you actually hit it.”
From the episode
Joshua Miller Discusses the Hot Hand Phenomenon (Podcast)