Simultaneous-Arrival Routing
Stagger departures so distributed actions reach every destination together
- Difficulty
- Expert
- Time to result
- ~months to results
- Steps
- 6
- Confidence
- 99%
The Thor router addressed a physical asymmetry: sending every order at once caused it to reach the closest exchange first. That early execution became a signal that could be used to cancel or reprice liquidity at farther venues. The solution reversed the usual assumption that simultaneous departure is fair. The team measured the path to each exchange, held back traffic on faster paths, and dispatched each message at a different time so all messages arrived within a narrow window. Because latency changes during the day, calibration was continuous rather than one-off. Katsuyama reports that the team reduced the arrival spread to roughly 290 microseconds and restored the ability to buy or sell what clients saw on their screens. The transferable mechanism is destination-aware synchronization.
Origin
Katsuyama's RBC team created the Thor router after discovering that equal-time dispatch reached BATS first and exposed orders before they arrived at other exchanges.
Core principles
- 01Equal send times do not produce equal arrival times
- 02Physical distance is part of system behavior
- 03The nearest destination can leak a signal to faster observers
- 04Synchronization must adapt as latency changes
- 05Success is measured by receiving the state that was visible when the action began
How to run it
- 1
Map the destinations
List every endpoint involved in the logical action and identify its physical location and network path. Treat geography as an operating variable.
Pro tip Measure the path actually used rather than relying only on straight-line distance.
Watch out A destination that looks nearby on a diagram may follow a slower network route.
- 2
Measure path latency
Collect repeated travel-time measurements for every destination. Establish both typical latency and its variation over time.
Pro tip Measure at several points during the operating day.
Watch out A single latency reading will become stale.
- 3
Set the common arrival target
Use the slowest required route to define the target arrival time. Calculate how long each faster route must wait before dispatch.
Pro tip Express success as a maximum arrival-time spread across all endpoints.
Watch out Optimizing every route for earliest arrival recreates the signal leak.
- 4
Stagger dispatch
Send traffic to the farthest or slowest endpoint first and progressively delay faster paths. Preserve the action as one synchronized event at arrival.
Pro tip Automate the delay schedule so operators do not manually time messages.
Watch out The order of departure should not be confused with the intended order of execution.
- 5
Validate the arrival window
Measure actual arrivals and compare them with the target spread. Test whether completion rates improve without introducing unacceptable delay.
Pro tip Compare the synchronized route with the previous concurrent route on the same outcome metric.
Watch out A technically narrow spread is useless if the desired transactions still fail.
- 6
Recalibrate continuously
Update delays as routes and congestion change. Monitor drift before it becomes large enough to expose the action again.
Pro tip Trigger recalibration when measured latency crosses a defined tolerance.
Watch out Static timing assumptions degrade in a changing network.
In the wild
RBC's original concurrent routing reached nearby BATS before more distant exchanges. Thor staggered dispatch so the orders arrived together, despite changing latency and physical distance. The team targeted an arrival window below 400 microseconds and achieved about 290 microseconds.
→ The router could buy or sell the quantity clients saw rather than filling at one venue and missing the rest.
Common mistakes
Sending every message together
Simultaneous departure produces sequential arrival when paths differ. The first arrival can expose the rest of the action.
Calibrating latency once
Katsuyama notes that latency changes throughout the day. A fixed delay schedule will drift away from synchronized arrival.
Ignoring the outcome metric
Timing precision matters because it restores fills. Optimizing microseconds without measuring completed execution misses the purpose.
Is it for you?
Best for
It is best for latency-sensitive distributed systems where one logical action must reach geographically separated endpoints together.
Not ideal for
It is not ideal when destinations can be handled independently or when latency varies too unpredictably to calibrate safely.
From the transcript
“if there is this geographical disparity, meaning you know, our order arrives at the exchange closest to us first, why not you know, stagger when…”
“Our goal was to have it done under four hundred micro seconds, and we got it down to about two d and ninety microseconds.”
“That product could actually now buy or sell what people saw on their screens.”
From the episode
IEX Group Brad Katsuyama: Masters of Business (Audio)
Brad Katsuyama