Audit the Audits
Consolidate past findings before launching another investigation.
- Difficulty
- Moderate
- Time to result
- ~weeks to results
- Steps
- 7
- Confidence
- 94%
Audit the Audits begins by treating prior reviews as a body of evidence rather than launching another isolated investigation. The team inventories previous audits, validates their findings, and translates overlapping recommendations into a common register. Each item is then cross-referenced against proof of implementation, partial action, rejection, or continued inaction. Unresolved findings are ranked by impact, recurrence, and connection to root causes. Only after this baseline is established does the team conduct new investigative work, targeting gaps that earlier reviews did not resolve. The output is a concise implementation map showing what was found, what changed, what remains outstanding, who owns it, and what should happen next. This focuses finite resources on outcomes instead of generating another report that can be ignored.
Origin
Warshaw proposed first conducting an audit of past MTA audits before commissioning another broad review of the transit authority.
Core principles
- 01Existing findings are evidence, not completed improvements.
- 02Finite audit resources should concentrate on consequential problems.
- 03Implementation status matters as much as producing recommendations.
- 04A consolidated baseline should precede another investigation.
- 05Root causes emerge when repeated findings are compared across time.
How to run it
- 1
Assemble the audit record
Collect relevant audits, reviews, responses, and implementation reports from the chosen period.
Pro tip Include internal, external, legislative, and regulator-produced reviews.
Watch out Missing older audits can conceal recurring findings.
- 2
Extract and normalize findings
Convert findings and recommendations into a common register, merging duplicates while preserving source references.
Pro tip Tag each finding by function, risk, and affected stakeholder.
Watch out Do not merge superficially similar findings with different root causes.
- 3
Validate the evidence
Confirm that each material finding was supported and remains relevant under current conditions.
Pro tip Record changed circumstances that make a recommendation obsolete.
Watch out Prior publication does not automatically make a finding correct.
- 4
Cross-reference implementation
Match each recommendation to documentary or operational evidence showing whether it was completed, partly completed, rejected, or ignored.
Pro tip Require demonstrated outcomes rather than accepting policy announcements.
Watch out A written response is not proof of implementation.
- 5
Prioritize unresolved risks
Rank outstanding findings by public impact, financial exposure, recurrence, and feasibility.
Pro tip Elevate clusters of repeated findings that point to a systemic cause.
Watch out Do not let the easiest recommendations displace the most consequential ones.
- 6
Investigate the gaps
Direct new audit work toward unresolved causes, disputed evidence, and areas not covered by previous reviews.
Pro tip Write a narrow question for each new workstream.
Watch out Launching another comprehensive audit can recreate the original fragmentation.
- 7
Track to closure
Publish responsible owners, milestones, evidence requirements, and follow-up dates for each priority action.
Pro tip Keep the consolidated register active after the report is released.
Watch out Without follow-up, the audit becomes another unimplemented document.
In the wild
Before commissioning another broad MTA audit, a controller's team gathers prior reviews, checks whether their findings remain valid, and maps each recommendation to evidence of implementation. Repeated unresolved findings then become the focus of narrower root-cause investigations.
→ Oversight resources shift from duplicative reviews to closing the transit authority's most consequential gaps.
A hospital network has undergone several cybersecurity assessments, yet similar access-control problems recur. An audit-of-audits register shows which controls were promised, which were deployed, and where implementation failed across facilities.
→ The next review targets the governance and ownership failures causing repeated control gaps.
Common mistakes
Counting reports as progress
Producing more audits creates activity but not improvement when prior recommendations remain unresolved.
Accepting self-reported completion
Organizations may label an action complete without evidence that the intended operational result was achieved.
Spreading resources too thin
Investigating many low-impact topics can prevent auditors from reaching the root causes of major failures.
Is it for you?
Best for
It is best for heavily reviewed agencies, programs, and organizations with fragmented audit histories.
Not ideal for
It is not ideal when no credible prior audits exist or an urgent new risk requires immediate investigation.
From the transcript
“I would first initially just do an audit of the audits.”
“make sure that those findings make sense, and then just simply ross reference to what have they implemented and what are still outstanding”
“before we launch into some new MTA audit, there have been so many MTA audits, let's just understand what of the findings have actually been…”
From the episode
BONUS: NY Comptroller Candidate Drew Warshaw on Changing the NY State Pension Funds
Drew Warshaw