Anecdote Credibility Test
Test stories for breadth, knowledge, relevance, and plausibility
- Difficulty
- Starter
- Time to result
- ~days to results
- Steps
- 5
- Confidence
- 93%
The test separates a useful illustration from evidence capable of supporting a general conclusion. First, count the cases: a single interview remains an N-of-one regardless of how compelling it sounds. Next, determine whether the people quoted possess direct, relevant knowledge and are describing their own experiences rather than repeating assumptions. Check whether the reporter asked a narrow question suited to those participants and whether several accounts point in the same direction. Finally, compare the story with broader data and apply a plausibility check for factual or internal inconsistencies. The output is not a binary judgment that anecdotes are worthless; it is a calibrated label showing whether a story merely illustrates a possibility or contributes credible support to a wider claim.
Origin
Extracted from Masters in Business, where Michael Hiltzik described his skepticism toward man-on-the-street reporting and unrepresentative inflation stories.
Core principles
- 01One person's experience is not a data series
- 02Interviewees should be positioned to know the subject
- 03Questions and selection methods affect the story
- 04Multiple relevant examples are stronger than one vivid case
- 05Plausibility is a filter, not proof
How to run it
- 1
Count the Cases
Determine how many independent people or examples support the narrative. Label a single case explicitly as an anecdote.
Pro tip Look for genuinely independent cases rather than several quotes drawn from the same household or network.
Watch out Multiple quotations do not necessarily constitute multiple independent observations.
- 2
Test Subject Knowledge
Ask whether each person is in a position to know the answer and has direct experience of the issue being discussed.
Pro tip Give more weight to precise firsthand details than to generalized impressions.
Watch out Confidence, status, or vivid language does not establish relevant knowledge.
- 3
Check Question Fit
Evaluate whether the topic is narrow enough for these interviewees to answer meaningfully and whether the visible questions could have shaped their responses.
Pro tip Prefer interviews where the participant's experience closely matches the claim being examined.
Watch out A conveniently available person may not be a relevant source.
- 4
Run a Plausibility Check
Look for factual errors, contradictions, or claims that conflict sharply with established evidence and ordinary experience.
Pro tip Investigate surprising discrepancies rather than automatically accepting or rejecting them.
Watch out Plausibility is only a screening device and can reflect the evaluator's own biases.
- 5
Classify the Evidence
Decide whether the anecdote is merely illustrative, suggestive of a pattern, or supported strongly enough by broader evidence to inform a general conclusion.
Pro tip State the classification when sharing the story so others understand its evidentiary weight.
Watch out Never turn an illustrative anecdote into a population-level claim without representative evidence.
In the wild
A report uses one family's grocery bill to claim that food costs doubled. A reader checks whether the family compared equivalent baskets, whether their quantities changed, and whether official price series and other households show a similar pattern.
→ The story is retained as a personal illustration but rejected as sufficient evidence for the broad inflation claim.
A founder hears one enthusiastic customer describe a new feature as essential. Before changing the roadmap, she interviews several users who perform the same task, verifies that they understand the feature, and compares their behavior with product analytics.
→ The founder distinguishes one customer's preference from a recurring market need.
Common mistakes
Generalizing From One Person
A vivid case can make a pattern feel common without showing how frequently it occurs. Keep the scope of the conclusion proportional to the evidence.
Interviewing Whoever Is Available
Convenience samples often include people who lack relevant knowledge or do not represent the population being discussed.
Using Experience as the Dataset
Personal experience can reveal a question worth investigating, but it should not automatically override systematic evidence.
Is it for you?
Best for
It is best for evaluating news reports, testimonials, customer interviews, and arguments built around individual experiences.
Not ideal for
It should not be used to dismiss personal testimony when the individual's experience is itself the subject under examination.
From the transcript
“a man on the street or a woman on the street that's that is that is an an equals one”
“you want to make sure that you know, if there's if there are interviews with individuals or families or couples, that there's more than one”
“they seem to actually know what they're talking about and they're talking about their own experiences, and that they sound plausible.”
From the episode
At The Money: Filtering Noisy News