Source-and-Series Credibility Audit
Validate every important number through provenance, arithmetic, context, and comparison.
- Difficulty
- Moderate
- Time to result
- ~weeks to results
- Steps
- 6
- Confidence
- 98%
The Source-and-Series Credibility Audit evaluates a number at four levels: provenance, internal coherence, contextual treatment, and external agreement. First, the analyst identifies the producer and returns to the original release rather than trusting a news interpretation. Next, the arithmetic, units, trend consistency, and suspicious outliers are inspected. The analyst then checks whether inflation, seasonality, methodology changes, or a truncated chart axis could distort the apparent result. Finally, the figure is compared with another credible source and interpreted in light of the producer's incentives. The output is not automatic acceptance or rejection, but a calibrated conclusion that distinguishes verified evidence, explainable anomalies, and unresolved claims requiring skepticism.
Origin
Extracted from Masters in Business.
Core principles
- 01Prefer original publications over second- or third-order interpretations.
- 02Judge sources by consistency, transparency, and incentives.
- 03Treat outliers as prompts for investigation rather than instant discoveries.
- 04Check arithmetic and contextual adjustments before accepting conclusions.
- 05Cross-check consequential figures against an independent source.
How to run it
- 1
Trace the provenance
Identify the organization that produced the figure and locate its original release, filing, or data series. Distinguish the producer from publishers interpreting it.
Pro tip Follow citations until reaching the first-party table or methodology note.
Watch out Repeated publication by reputable outlets does not make a secondary interpretation primary evidence.
- 2
Check the arithmetic
Recalculate important totals, rates, or comparisons where possible. Confirm units, denominators, and periods before assessing the conclusion.
Pro tip Reproduce the calculation in a small spreadsheet or notebook.
Watch out A plausible-looking result may still use mismatched periods or denominators.
- 3
Inspect coherence and outliers
Compare the figure with its surrounding trend and look for unusual jumps, breaks, or contradictions. Investigate anomalies rather than dismissing or celebrating them immediately.
Pro tip Plot a sufficiently long series to expose whether the latest movement is genuinely unusual.
Watch out An outlier can reflect a methodology change or data error rather than a real-world event.
- 4
Restore missing context
Check whether nominal values need inflation adjustment, whether recurring patterns require seasonal adjustment, and whether methodology has changed. Inspect chart axes for visual exaggeration.
Pro tip Compare nominal and real versions of the same series when both are available.
Watch out A truncated vertical axis can make a small change appear dramatic.
- 5
Cross-check independently
Compare the result with another credible source or related series. Determine whether any disagreement comes from definitions, timing, or a substantive flaw.
Pro tip Use a source with different incentives or collection methods when possible.
Watch out Two websites may appear independent while reproducing the same underlying data.
- 6
Account for incentives
Consider whether a company, trade group, think tank, or politician benefits from a particular interpretation. Attribute interested claims and carry the limitation into the conclusion.
Pro tip Separate the organization's raw figure from the narrative it attaches to that figure.
Watch out Bias does not automatically falsify data, but it raises the verification burden.
In the wild
A report cites a trade association's housing figure as proof of a strong market. The analyst retrieves the original release, checks the comparison period and seasonal treatment, plots the longer trend, and compares it with a government series. The final note attributes the association's figure and explains the definitional difference instead of repeating its interpretation as neutral fact.
→ Readers receive a qualified, independently checked account of the market rather than industry spin.
A viral chart appears to show an unprecedented jump in spending. The analyst discovers that the values are nominal, the vertical axis begins near the prior low, and the latest point is a seasonal peak. After inflation adjustment, seasonal comparison, and restoration of a zero baseline, the movement is far less dramatic.
→ A visually persuasive but misleading claim is corrected before it influences an investment decision.
Common mistakes
Trusting interpretation over source
News reports can add errors, politics, or missing context that are absent from the original release.
Treating every outlier as insight
An unusual value warrants checking because it may arise from methodology, transcription, or calculation problems.
Ignoring the producer's incentives
Interested organizations may publish accurate inputs while steering readers toward a self-serving conclusion.
Is it for you?
Best for
It is best for people evaluating economic releases, company claims, charts, or statistics supplied by interested organizations.
Not ideal for
It is not ideal when a low-stakes decision must be made immediately and the original evidence is genuinely unavailable.
From the transcript
“I look for outliers in the data. And when I see something like that, it warrants further checking, and it warrants skepticism.”
“if I can, I check one source against the other and then try to see if doing that turns up some flaw or flaws in…”
“I will always try to go back to the original print, not rely on somebody's interpretation.”
From the episode
At The Money: How to Manage Your News & Data Flow