Controlled Market-Structure Experiment
Turn suspicious execution patterns into evidence with isolated, repeatable trades
- Difficulty
- Advanced
- Time to result
- ~weeks to results
- Steps
- 6
- Confidence
- 98%
Katsuyama's team moved from frustration to explanation by treating poor order execution as an experimental problem. They first observed a repeatable trigger: displayed offers disappeared only after an order was sent. With RBC's approval, they used the firm's own capital, imposed a loss ceiling, and generated data by buying and selling rather than experimenting with client orders. They then varied the routing pattern, testing exchanges separately before adding multiple destinations. Single-exchange orders filled, while simultaneous orders reached the nearest venue and missed farther venues. Specialists such as Ronan Ryan supplied the geography and network-latency knowledge that traders lacked. The method combines controlled variation, bounded risk, repeated observations, and cross-functional interpretation to expose a hidden mechanism in a complex system.
Origin
Katsuyama developed the approach at RBC after traders repeatedly saw displayed stock vanish when they attempted to buy it and colleagues dismissed the pattern as coincidence.
Core principles
- 01Persistent anomalies deserve controlled tests rather than dismissal
- 02Change one interaction pattern at a time
- 03Use owned capital instead of exposing clients to experimental risk
- 04Collect enough observations to distinguish mechanism from coincidence
- 05Combine partial expertise until the whole system becomes visible
How to run it
- 1
Define the anomaly
Describe what changes, when it changes, and what action appears to trigger it. Establish that the behavior affects more than one isolated user.
Pro tip Delay the triggering action at random intervals to test whether the system remains stable until you act.
Watch out Do not accept coincidence as an explanation without testing it.
- 2
Bound the experiment
Secure authorization, choose capital that can legitimately bear the test, and set a maximum loss. Keep client activity outside the experiment.
Pro tip Frame the expense as the cost of acquiring decisive data points.
Watch out Never use client orders merely because they are readily available.
- 3
Test components in isolation
Interact with one destination or subsystem at a time and record the result. Use the isolated result as the baseline for later comparisons.
Pro tip Keep price and order characteristics comparable across tests.
Watch out Testing the full system first can hide which component produced the effect.
- 4
Add interactions progressively
Repeat the test while adding two, three, and then more destinations. Track how fill rates or outcomes change with system complexity.
Pro tip Pay special attention to which component succeeds first and which components fail afterward.
Watch out Aggregate outcomes can conceal a consistent sequence across components.
- 5
Map the physical mechanism
Bring in specialists who understand networking, architecture, or another missing domain. Translate the observed sequence into distances, latency, and signals.
Pro tip Ask each expert to contribute the part of the system they know rather than expecting one person to know everything.
Watch out A domain expert may possess crucial knowledge without recognizing its value in your context.
- 6
Check the explanation against new data
Use the proposed mechanism to predict a new result, then run another controlled test. Keep the explanation only if the observed outcome follows the prediction.
Pro tip Prefer a mechanism that explains both the successful and failed interactions.
Watch out A compelling story is not a substitute for a repeatable prediction.
In the wild
Katsuyama's team sent orders to exchanges one at a time and received the displayed amount. As they added exchanges simultaneously, fill rates fell: they continued to receive stock at BATS, the physically closest venue, while missing stock elsewhere. Hundreds of millions of shares supplied repeated observations, and offsetting purchases and sales made the experiment close to a financial wash.
→ The team isolated a latency signal that faster traders could detect at one exchange and race to other venues.
Common mistakes
Calling a repeatable pattern coincidence
A label does not explain why the market remained unchanged until the trader pressed the button. The trigger needed a controlled test.
Changing every variable at once
Testing all destinations together obscures the baseline. Isolation reveals whether each component behaves correctly on its own.
Expecting one expert to see the whole system
The traders understood execution while Ryan understood physical networks. The mechanism appeared only when their partial knowledge was combined.
Is it for you?
Best for
It is best for teams investigating repeatable failures in complex systems where several components interact at different speeds.
Not ideal for
It is not ideal when experiments would expose clients to unapproved risk or when the event cannot be reproduced safely.
From the transcript
“In order to test these theories out, we had to set up a series of experiments interacting with different exchanges in different ways.”
“We said, okay, we'll just pick one exchange, one amount. We'd always get a hundred percent.”
“No one knew the whole picture. We kind of like put the jig saw together.”
From the episode
IEX Group Brad Katsuyama: Masters of Business (Audio)
Brad Katsuyama