The Same Matching Engine Measured 130,000/s and 106/s

One article reports millions of operations per second. Another production service reports hundreds of commands per second. Both can be honest, because they are measuring different systems and calling the result the same thing. We ran into this inside a single codebase — here are four measurements of one matching engine, taken the same afternoon on the same machine:

That is a spread of roughly 1,200x between the top and bottom rows. If you publish only the first number you have a marketing asset. If you publish only the last you have understated your engine by three orders of magnitude. Neither tells an operator what to buy or a developer what to fix.

So every result here starts with a boundary statement. “Engine-only” means deterministic book mutation and nothing else. “Conductor” adds validation, sequencing, and the durable state commit. “Confirmed pipeline” adds event publication with broker confirms. “Client round trip” would add the network edge and the response path — we have not measured that one honestly yet, so it is not in the table.

Matching cost depends entirely on order behaviour, so the workload matters more than the hardware. Ours alternates equal-price, equal-volume limit bids and asks on a single pair. Every two commands fully match. The book ends at zero depth.

This is the maximum-contention path for a single-writer engine, and it is chosen on purpose. It exercises the hot loop with no parallel-pair relief, produces two events per command, and leaves nothing resting to inflate the numbers. It is the opposite of the benchmark that posts limit orders at unique prices and never matches anything — that one measures your hash map.

It also means these results do not generalise upward. Real traffic has multiple pairs, which our partitioned design handles in parallel, so a realistic mixed workload should do better on aggregate throughput and worse on nothing. But we have not run that test yet, so we are not claiming it.

• Publish the command-type mix and the fill-count distribution, not just “orders per second”.
• State the book shape: markets, active orders, occupied price levels, orders per level, tick range, concentration near the touch.
• Warm the runtime and the book before recording. JIT and allocator behaviour differ in the first thousand commands.
• Use deterministic IDs, timestamps, and seeds so two implementations receive byte-identical work.
• Run long enough to hit garbage collection and persistence cycles. A three-second sample measures the gaps between them.
• Verify final state and an event digest after every run — see below for why.

The gap between 380 commands/s at the conductor and 106 through the full pipeline is the interesting one, and it is not subtle once you look. The outbox published and confirmed events strictly sequentially to preserve partition order. Correct, and it capped one worker at 517 events/s draining a 10,000-event backlog.

This workload produces two events per command. 517 events/s is therefore about 258 commands/s of headroom before anything else — and once the conductor and outbox competed for the same Redis and RabbitMQ, the combined figure fell to 106.

The contention shows up clearly in the broker timings. RabbitMQ accepted and confirmed each 5,000-command producer burst in 296–303ms while the outbox was idle. In the concurrent run the same burst took 1,088ms — 3.6x slower — while producer-side call latency stayed low:

p50   0.029 ms
p95   0.342 ms
p99   2.590 ms
max  12.296 ms

That distribution is what a healthy producer in front of a saturated consumer looks like: publishing is cheap, the work behind it is not. It is also why measuring only the client-visible submit latency would have told us the system was fine.

The fix follows directly from the measurement — a bounded confirm window: publish a contiguous batch on one channel in stream order, wait for all confirms, and advance the stream acknowledgement as one batch. Ordering is preserved because sends still go out in stream order on a single channel; throughput improves because you are no longer paying a full round trip per event. That change is implemented and awaiting a fresh live baseline. We are not quoting a number for it until we have one.

Dragonfly is a multithreaded, Redis-compatible datastore, and the multithreaded part is the headline. We migrated to it and then benchmarked it against plain Redis 7 on the transaction shape the conductor actually issues — HSET, ZADD, and XADD inside one MULTI.

Dragonfly was 30–31% slower across both connection counts. Not catastrophically slower, and not a criticism of Dragonfly — a transaction-heavy, small-payload, single-partition workload is close to the worst case for a design whose advantage comes from sharding work across cores. Our transactions all target one hash tag by construction, which is exactly the shape that does not shard.

The end-to-end conductor runs against Dragonfly came in between 164 and 344 commands/s with zero failures and exactly the expected event count. That spread is too wide to draw a datastore conclusion from — local RabbitMQ and Docker Desktop scheduling dominate at that scale, which is itself a useful reminder that end-to-end numbers are the wrong instrument for component comparisons.

The architectural bet is that matching scales horizontally through independent cells — one writer, one storage group, one outbox per partition. So we tested it: same synthetic transaction, one cell versus two.

Both cells shared one Docker Desktop CPU and disk budget. Adding a second cell added processes, not CPU, not disk bandwidth, and not a failure domain — the three things that would have made it faster. The result does not show that partitioning fails. It shows the test did not test partitioning.

This is the most common way scaling benchmarks lie, and it is easy to do accidentally. The architecture claim and the physical experiment have to match. If your design says “independent failure domains give independent capacity”, the experiment must actually provide independent resources, or all you have measured is contention you introduced yourself.

The gate we set for the real test: cells on separate hosts or with enforceable CPU and storage budgets, independent market traces routed through the production function, at least 1.5x aggregate sustainable throughput going from one cell to two, per-partition p99 reported separately so one hot market is not hidden by idle ones, zero ownership conflicts, no sequence gaps, and equivalent final-state digests. Until then the multi-cell design is a hypothesis.

Memory-only state, an asynchronous replica, local append-only persistence, and local-plus-replica fsync are four different products that get described with the same word. In the table above, the only difference between 25,621 tx/s and 3,254 tx/s is the durability policy.

That is what a real recovery point objective costs on this hardware. The conductor issues WAITAOF 1 1 after the atomic transaction and before acknowledging the command to RabbitMQ, so a confirmed command has been fsynced on both primary and replica. If the datastore cannot satisfy it within the timeout, the command fails rather than silently succeeding with weaker semantics.

We had also, at one point, changed this without noticing. Replacing KeyDB’s every-second AOF with Dragonfly’s five-minute snapshot moved the worst-case unreplicated loss window from one second to five minutes. That arrived attached to a datastore migration, not as a durability decision, which is exactly how these things happen. It is now a named policy that the process refuses to start without.

• Label the exact policy alongside every throughput number. “Persistent” is not a policy.
• Measure journal or AOF acknowledgement latency separately and in combination with the command.
• Record durability timeouts and replica availability during the run — a fast run with an absent replica is measuring memory mode.
• If the service fails closed when durability is unavailable, exercise that path under load, not just in a unit test.
• Disk settings, filesystem, host cache, storage class, and virtualisation all move this number. A laptop tmpfs benchmark cannot support a production RPO claim.

A benchmark that only checks elapsed time will happily reward dropped work, duplicate execution, broken FIFO, negative remainders, and unconfirmed events. Every run here asserts the invariants and reports them alongside the throughput: zero command failures, zero outbox retries, zero dead letters, the exact expected event count, and zero remaining book depth.

That last one is a nice property of the alternating workload — because every two commands fully match, any residual depth at the end means something did not execute that should have, and it is visible without a digest.

• Compute a deterministic digest of the final books and the ordered event stream, and compare it across implementations.
• Assert executed buy quantity equals executed sell quantity, no order fills beyond its accepted quantity, every live order appears in exactly one side and price level, empty levels are removed, and FIFO holds within a level.
• Count accepted, rejected, duplicate, rested, cancelled, and filled commands. Duplicates caught by dedup are a success metric, not an error.
• For replay tests, run the trace uninterrupted and with injected crashes, then compare. Kill before commit; after commit but before broker ack; after publication but before outbox ack.
• A correct idempotent design reaches the same authoritative state in all three cases. At-least-once delivery may repeat an event ID — prove the consumer applies it once.

Zero reported failures means nothing if the harness never checked. Say what you asserted, not just that nothing went wrong.

Partway through, we reset the isolated Redis matching namespace while the outbox was still running. That removed the outbox’s stream consumer group.

The worker retried the resulting NOGROUP error forever, and its metrics endpoint started returning HTTP 500 — so the component was both stuck and unable to tell anyone it was stuck. In production that is a silent event-delivery outage with a broken health signal on top, which is roughly the worst combination available.

The outbox now recreates a missing consumer group automatically, and its pending metric returns zero while the group is being restored rather than erroring. We discarded the affected run and reran it after rebuilding.

The general lesson is worth more than the specific fix: benchmarks are fault injection with a stopwatch attached. Sustained load plus a hostile hand on the infrastructure finds recovery gaps that unit tests structurally cannot, because unit tests do not delete things out from under a running process. Budget time in every performance run to break something on purpose.

A single end-to-end timer tells you the request is slow. It does not tell you which capacity to buy or which code to change. Instrument the stages separately: queue wait, validation, candidate lookup, match calculation, transaction execution, durability acknowledgement, event-stream delay, broker confirm, consumer application.

Correlate every command and event with partition, pair, command ID, event ID, and sequence — and keep the detailed logging out of the hottest comparison loop, or you will be profiling your logger.

Then change one constraint at a time and re-run the same trace: batch candidate reads, cut redundant serialisation, reuse connections, bound the publisher-confirm window, improve data locality, isolate disk. Compare percentiles and invariants, not just the headline.

A result without an environment is an anecdote. Ours, in full:

Host        Apple M3 Max, 14 logical CPUs, 36 GiB RAM, macOS arm64
Engine      Docker Desktop 28.1.1
Runtime     Node.js 20.20.2 (containers)
Broker      RabbitMQ 3.9 Management
Datastore   Redis 7 Alpine / Dragonfly 1.39.0 / Valkey + AOF
Workload    single pair, alternating equal-price bid/ask, ~2 events per command
Consumers   durable benchmark sink queues bound; no downstream business consumers

node scripts/matching-domain-performance.js 100000    # pure domain
node scripts/conductor-performance.js 1000            # conductor
BENCH_WAIT_OUTBOX=1 node scripts/conductor-performance.js 5000
node scripts/outbox-drain-performance.js              # outbox backlog

BENCH_REDIS_URL=redis://localhost:6379 pnpm perf:redis -- 20000 16
BENCH_REDIS_URLS=redis://127.0.0.1:6580,redis://127.0.0.1:6581 \
  BENCH_WAITAOF=1 pnpm perf:redis:cells -- 10000 16

And the things this does not measure, stated plainly: no network latency, no downstream consumer or database work, one hot pair, no resource limits, no multi-pair parallelism, no client round trip. Docker Desktop loopback differs materially from deployment conditions in every one of those dimensions.

The operational conclusion is more useful than any single figure. Confirmed event delivery was the immediate end-to-end bottleneck; the matching arithmetic had three orders of magnitude of headroom; durability costs roughly 8x and is worth it; independent cells are the intended scale unit and shared-host tests cannot demonstrate that. “Make matching parallel” was the wrong task. “Reduce round trips and open a bounded confirm window” was the right one — and only the boundary-by-boundary measurement made that visible.