One Thousand Sample Comparison
Validate a new diagnostic platform against the incumbent at realistic scale
- Difficulty
- Advanced
- Time to result
- ~months to results
- Steps
- 5
- Confidence
- 98%
The One Thousand Sample Comparison is a controlled migration method for diagnostic technologies. A laboratory processes the same large, representative sample set with both its incumbent platform and a proposed replacement. Rather than treating agreement as sufficient, the team isolates every conflicting result and applies an independent third technology to determine which platform was correct. The larger sample size matters because comparisons involving only one hundred samples can show perfect agreement while missing error modes that appear only a few times per thousand. The resulting evidence supports a defensible decision about accuracy, throughput, and operational adoption. Publishing the findings adds external scrutiny and gives clinicians, regulators, and laboratory leaders greater confidence when moving from an established technology to a newer one.
Origin
Charles Strom developed the comparison at Quest Diagnostics while replacing low-volume genetic-testing technologies with platforms capable of accurate, cost-efficient, high-throughput operation.
Core principles
- 01Validate technology at production-relevant scale
- 02Treat every discrepancy as evidence requiring resolution
- 03Use an independent method to adjudicate conflicting results
- 04Replace incumbents only after demonstrating superior accuracy
- 05Publish validation evidence to build confidence
How to run it
- 1
Define the migration decision
Specify the incumbent platform, proposed replacement, intended clinical use, and performance requirements. Include accuracy, throughput, and cost-efficiency criteria.
Pro tip Choose samples that represent routine cases as well as uncommon variants and edge cases.
Watch out A convenient but unrepresentative sample set can conceal production failures.
- 2
Run one thousand paired samples
Process the same one thousand samples independently with the old and new technologies. Preserve complete results and quality-control data from both runs.
Pro tip Blind operators to the other platform's results where practical.
Watch out Do not substitute a hundred-sample pilot for the full comparison.
- 3
Isolate discrepancies
Compare the paired outputs and enumerate every disagreement, even when the total discrepancy rate appears small.
Pro tip Classify discrepancies by sample type, variant, instrument run, and failure mode.
Watch out Aggregate agreement percentages can obscure clinically important errors.
- 4
Adjudicate independently
Retest each discrepant sample with a third technology that does not share the same core failure mode. Use that result to determine which original platform was correct.
Pro tip Predefine how inconclusive adjudication results will be handled.
Watch out Choosing an adjudicator too similar to one platform can bias the decision.
- 5
Decide and document
Use the adjudicated evidence to select the stronger platform, document limitations, and publish or internally review the findings before migration.
Pro tip Pair accuracy evidence with throughput and cost analysis before rollout.
Watch out Do not assume newer automatically means better.
In the wild
At Quest Diagnostics, Strom's team compared one thousand samples using an older genetic-testing technology and a proposed newer platform. Although smaller hundred-sample comparisons often showed complete agreement, the expanded set exposed several discrepancies. A third technology was then used to determine which platform had produced the correct result.
→ The team demonstrated that the new technology outperformed the old one and migrated with greater confidence.
A hospital considering a new sequencing pipeline runs one thousand archived clinical samples through both pipelines. Five disagreements emerge, including two rare variants absent from its pilot. Independent confirmatory testing shows that the new pipeline is correct in four cases but misses one clinically significant variant, prompting remediation before launch.
→ The laboratory fixes a hidden failure mode before exposing patients to it.
Common mistakes
Validating on only one hundred samples
A small comparison can produce complete agreement while missing errors that occur only a few times per thousand samples.
Counting discrepancies without resolving them
Raw disagreement reveals a problem but does not establish which platform is correct; an independent third method is essential.
Evaluating accuracy without operational fit
A clinically accurate platform still needs adequate throughput and cost-efficiency for routine laboratory use.
Is it for you?
Best for
It is best for clinical laboratories evaluating a new assay, instrument, algorithm, or high-throughput diagnostic platform.
Not ideal for
It is not ideal for early exploratory research where representative samples and an independent adjudication method are unavailable.
From the transcript
“we invented something called a one thousand sample comparison”
“before we would introduce a new platform, we would look at a thousand samples with the old technology and the new technology.”
“if there were any discrepancies, we would resolve that discrepancy with a third technology”
From the episode
Charles Strom on the Diagnostics Industry
Charles Strom