Test Conventional Wisdom
Turn received wisdom into testable claims and search for disconfirming data
- Difficulty
- Easy
- Time to result
- ~weeks to results
- Steps
- 7
- Confidence
- 96%
Test Conventional Wisdom is a repeatable research habit for converting received ideas into falsifiable questions. Whenever a teacher, authority, industry narrative, or established model presents a claim, ask whether it is true and whether anyone has tested it. Define the expected observation, gather suitable data, and compare the claim with credible alternatives. The objective is not reflexive contrarianism: Arnott says many tests confirm conventional wisdom. The valuable cases arise when a widely held belief fails empirical scrutiny, because decisions, products, or prices may still reflect the false assumption. Researchers must also anticipate resistance. Evidence that overturns a professional worldview can provoke anger from people whose careers or identities depend on the old premise, so criticism should be treated as a predictable social response rather than proof that the test was wrong.
Origin
Arnott describes persistent empirical skepticism as a foundation of his career. He repeatedly tested financial conventional wisdom, published contradictory findings, and eventually learned to expect resistance when evidence challenged established beliefs.
Core principles
- 01Authority does not make a claim true
- 02Convert beliefs into testable propositions
- 03Seek disconfirming as well as confirming evidence
- 04Accept conventional wisdom when data supports it
- 05Investigate false consensus for opportunity
How to run it
- 1
Capture the claim
Write the conventional belief as a specific statement rather than a vague slogan.
Pro tip Preserve the original wording before translating it into variables.
Watch out A claim that can explain every possible outcome cannot be tested.
- 2
Define falsifying evidence
Specify what observation would count against the claim and what result would support it.
Pro tip Set criteria before looking at the results.
Watch out Changing the standard after seeing the data creates confirmation bias.
- 3
Gather relevant data
Collect data spanning appropriate periods, populations, and market conditions.
Pro tip Include inconvenient periods and failed cases, not only famous successes.
Watch out A narrow or survivorship-biased sample can validate a false belief.
- 4
Run competing tests
Test both the conventional explanation and plausible alternative mechanisms.
Pro tip Use simple tests first, then add complexity only when it changes the inference.
Watch out Correlation alone may not identify the operative mechanism.
- 5
Classify the result
Record whether the belief is supported, rejected, or remains uncertain, including limitations.
Pro tip Treat confirmation as a useful result rather than a failed attempt at contrarianism.
Watch out Do not overstate a noisy or context-dependent finding.
- 6
Search for practical implications
When a widely held claim appears false, identify decisions or prices still anchored to it.
Pro tip Start with small, reversible applications of the finding.
Watch out Being correct does not ensure immediate adoption or profit.
- 7
Prepare for resistance
Separate substantive criticism from emotional opposition caused by a threatened worldview.
Pro tip Invite replication and publish assumptions transparently.
Watch out Dismissing every critic as defensive can blind you to genuine flaws.
In the wild
Rather than accepting that market-cap weighting was the neutral default, Research Affiliates tested portfolios weighted by sales, book value, profits, cash flow, and employment. The recurring result suggested that breaking the price-weight link generated a systematic advantage.
→ A challenged convention became a practical investment methodology.
After Arnott published a paper titled Death of the Risk Premium, an investor later admitted hating him because the paper challenged everything he believed. Years later, the investor concluded that Arnott had been right.
→ Arnott learned to treat emotional resistance as a predictable consequence of overturning established assumptions.
Common mistakes
Practicing automatic contrarianism
The goal is to test beliefs, not to assume consensus is always wrong. Arnott says many tests confirm the prevailing view.
Testing after choosing the answer
Searching only for supportive evidence turns skepticism into motivated reasoning.
Confusing outrage with refutation
A threatened audience may react emotionally, but the validity of the finding depends on evidence and replication.
Is it for you?
Best for
It is best for researchers, analysts, and professionals working in fields where data can test widely repeated claims.
Not ideal for
It is not ideal where evidence is unavailable, experiments would be unethical, or findings cannot be separated from major confounding factors.
From the transcript
“Whatever you're taught from whoever teaches it to you, ask the question, um, is this true? Have people tested it?”
“If there's a bit of conventional wisdom floating around, I love to test it, and half the time I find it's absolutely correct, and half…”
“It's it's a given. If you up end somebody's worldview, they're going to be angry because they've built their career on the basis of a…”
From the episode
Robert Arnott on Global Asset Management (Podcast)
Robert Arnott