Say-Do Consistency Test for Manager Selection
Select managers whose claimed process matches the source of their results
- Difficulty
- Expert
- Time to result
- ~months to results
- Steps
- 6
- Confidence
- 98%
Vanguard's manager-selection method tests whether a manager's explanation of their edge matches what the portfolio actually does. The evaluator first records the claimed process, then uses attribution analysis to identify whether returns came from stock selection, sector exposure, or another factor. Outperformance is not automatically positive evidence: if a self-described bottom-up stock picker wins only through sector allocation, the result raises a flag. Average performance can be more credible when the claimed source of skill is visibly adding value despite factor headwinds. The review then goes deeper into how ideas enter and leave the portfolio, alongside lower turnover, talent retention, process consistency, next-generation development, and succession. Repeated meetings and senior-management involvement make the assessment continuous rather than a one-time ranking exercise.
Origin
McNabb describes Vanguard's global search for subadvisors and the rigorous portfolio-review process used to oversee 31 firms running about 75 mandates.
Core principles
- 01Low cost reduces the drag active managers must overcome.
- 02An investment process should be explainable and consistently applied.
- 03Performance is only useful evidence when its source matches the claimed process.
- 04Talent retention and succession determine whether a process can persist.
- 05Senior leaders should spend time on the decisions central to the business.
How to run it
- 1
Capture the claimed process
Ask the manager to explain how decisions are made and what should drive results. Make the claim specific enough to test against the portfolio.
Pro tip Document whether the manager claims stock selection, factor exposure, sector allocation, or another source of advantage.
Watch out A vague process cannot be verified and can be rewritten after the result.
- 2
Attribute the result
Break performance into its actual drivers. Determine whether value came from the decisions the manager says are central to the process.
Pro tip Inspect both strong and average periods because factor headwinds can obscure evidence of skill.
Watch out Positive performance for the wrong reason is a warning, not confirmation.
- 3
Test say-do consistency
Compare the stated process with the attributed result and portfolio behaviour. Reward alignment and investigate material mismatches.
Pro tip Treat consistency as evidence accumulated over time, not a conclusion from one period.
Watch out Do not let the size of an outperforming number override a process mismatch.
- 4
Inspect the decision pipeline
Meet the analyst and portfolio teams to learn how ideas enter, change, and leave the portfolio. Examine the working process behind the public description.
Pro tip Ask for recent examples of accepted and rejected ideas.
Watch out A polished senior-level explanation may not reflect how the wider team actually works.
- 5
Assess organizational durability
Review turnover, talent retention, process discipline, development of the next generation, and succession. Decide whether the source of skill can survive personnel changes.
Pro tip Give the people side equal weight when a strategy depends on a stable team.
Watch out A first-generation track record does not prove the firm can transfer its process.
- 6
Review continuously
Bring managers back for regular, rigorous reviews involving the people accountable for the allocation. Use the meetings to update evidence rather than merely confirm the existing choice.
Pro tip Keep senior leadership involved because manager selection is central to the active business.
Watch out A one-time selection process cannot detect later process drift or succession risk.
In the wild
A manager says bottom-up stock selection drives the portfolio, but attribution shows no contribution from stock selection and all outperformance comes from sector selection. Vanguard would flag the mismatch even though the headline return is positive.
→ The evaluator avoids mistaking an unintended exposure for evidence of the claimed skill.
A bottom-up manager has only average overall performance during a period of factor headwinds, but attribution shows that stock selection added value. The evidence reinforces the manager's account of how the process works even though the headline result is not exceptional.
→ The evaluator separates process evidence from market conditions that temporarily affect the total return.
Common mistakes
Selecting on headline returns
A strong number can come from a different exposure than the manager claims, making it weak evidence of repeatable skill.
Ignoring organizational continuity
Talent loss, inconsistent process, and weak succession can destroy an otherwise credible investment approach.
Delegating the core review
McNabb argues that senior management time is justified because selecting managers is the most important task on Vanguard's active side.
Is it for you?
Best for
It is best for institutions, fund sponsors, or allocators selecting and monitoring external active managers.
Not ideal for
It is not suitable for choosing a manager from short-term headline returns without access to process, people, and attribution evidence.
From the transcript
“what we're really looking for is an explainable process that's very consistent.”
“you may be happy that you have two d basis points about performance, but you have it for the wrong reason.”
“what we really want to see is a consistency between what they say and what actually happens.”
From the episode
Vanguard CEO Bill McNabb: Masters in Business (Audio)