We Deleted the Distributed Lock From Our Matching Engine

The original design serialised matching per trading pair with a Redis lock. It is the obvious thing to reach for, and the implementation was the textbook one:

async acquireLock(key: string, ttl = 5000) {
  const result = await this.redis.set(
    `lock:${key}`, Date.now().toString(), 'PX', ttl, 'NX',
  );
  return result === 'OK';
}

async releaseLock(key: string) {
  await this.redis.del(`lock:${key}`);   // unconditional
}

The release is unconditional. It deletes whatever is at that key, not the lock the caller acquired. Most of the time those are the same thing. Here is the interleaving where they are not:

1. Writer A acquires lock:BTC-USD with a 5-second TTL and starts matching.
2. A’s command sweeps a deep book — or hits a GC pause, or a Redis stall, or an event-loop block from an unrelated handler. It takes longer than 5 seconds.
3. The lock expires. Nobody is told; expiry is silent by design.
4. Writer B acquires lock:BTC-USD cleanly and starts matching the same book.
5. Writer A finishes and calls releaseLock — deleting B’s lock while B is still working.
6. Writer C acquires the now-free lock. B and C are now both mutating one order book.

Step 6 is where price-time priority stops being a property of the exchange and starts being a property of the scheduler. Two writers reading the same best ask can both decide to fill it. Nothing in the code detects this; you find out from a reconciliation mismatch, or from a customer.

The standard fix is a fencing token: store a unique owner value, and release with a Lua compare-and-delete so you only remove your own lock. That closes step 5. It does not close step 3 — A can still be mid-match with an expired lock while B legitimately holds a new one. To close that you need A to re-check ownership before every write, which means the lock has to be part of the transaction, which means you have rebuilt a worse version of what the queue already gives you for free.

Instead of coordinating writers, we stopped having more than one. Commands for a pair route to a partition queue; the queue permits exactly one active consumer; that consumer is the only thing allowed to mutate the book. RabbitMQ enforces this, and it is roughly four lines of configuration.

@RabbitRPC({
  exchange: MATCHING_COMMAND_EXCHANGE,
  routingKey: matchingCommandQueue(ASSIGNED_PARTITION),
  queue: matchingCommandQueue(ASSIGNED_PARTITION),
  queueOptions: {
    durable: true,
    arguments: {
      'x-queue-type': 'quorum',
      'x-single-active-consumer': true,
    },
  },
})
processCommand(payload: unknown): Promise<ProcessedCommandResult> {
  return this.service.processCommand(payload);
}

Single active consumer means only one registered consumer receives messages at a time; if it disconnects, another takes over. RabbitMQ’s own documentation recommends it precisely when messages must be processed in arrival order. A quorum queue adds replicated queue state, so the ordering guarantee survives a broker node failure rather than a broker restart.

It is worth being precise about why this is better rather than just different:

• Ordering becomes a broker property instead of a timing property. There is no TTL to tune, renew, or lose a race against.
• There is no stale owner capable of deleting a new owner’s lock, because there is no lock.
• Backpressure becomes visible as queue depth instead of invisible as lock contention.
• Recovery is “resume from the last acknowledged command,” which is a thing the broker already knows how to do.
• Scaling becomes deliberate: you add partitions, not competing writers for the same pair.

The trade-off is honest and worth stating: one hot pair is now limited to one consumer. That is the correct constraint for a price-time book. A pair needs a serialisation point somewhere, and it is better to have it be an explicit, observable queue than an accidental, invisible lock.

The important property is exclusive ownership of the partition state, not any particular OS primitive. The owner can be a thread, an actor, a process, an event-loop task, or an isolated service. Inside a Node.js process, ours is a promise chain: every command appends to a tail, so command N+1 cannot start until command N settles, success or failure.

private processingTail: Promise<void> = Promise.resolve();

async processCommand(value: unknown): Promise<ProcessedCommandResult> {
  const result = this.processingTail.then(() =>
    this.processCommandSerially(value),
  );
  // Never let a rejection break the chain for the next command.
  this.processingTail = result.then(() => undefined, () => undefined);
  return result;
}

This is a belt-and-braces measure — the broker already guarantees one in-flight command — but it costs nothing and it means the invariant holds even if someone later attaches a second command source, or a health check starts calling into the service directly.

Process isolation is usually easier to operate than shared memory. A crash takes down fewer markets, memory growth is visible per cell, and a standby can pick up an ordered queue without any handoff protocol you had to write. You pay for it in serialisation and network hops between stages. Shared-memory threads cut copying but move correctness into synchronisation inside a single failure domain. Pick the boundary your team can observe, recover, and test — nanoseconds saved inside an opaque concurrency scheme are worthless during an incident where nobody can prove which order won.

Take two aggressive buys reaching the same ask level with two matching threads running. Both threads read the same best ask and the same remaining quantity. To avoid overfilling the maker they need coordination; to preserve arrival order they need more; to compute fees from consistent state, publish one authoritative sequence, and keep the cancellation index correct they need more still. A mutex around that critical section makes the matching serial again, just with extra steps.

Finer locks split the book into regions — until an order crosses several price ranges and acquires several regions, and you inherit lock ordering, retry, and starvation. Optimistic concurrency moves the cost into conflict detection: it looks great under light traffic and collapses exactly at the best price under a burst, which is when it matters.

And there is a correctness trap underneath the performance one. A retried command must not silently receive worse time priority than it originally earned. Any design that lets scheduling accidents leak into execution priority is wrong no matter what its ops/sec graph says. A benchmark that omits contention, cancellations, partial fills, and invariant checks will happily reward the design that is least predictable under real load.

Independent markets are the natural unit. BTC-USD and ETH-USD share no price-time priority, so their books can advance simultaneously. Route each pair to a logical partition with a stable function, and give each partition its own queue, conductor, storage cell, and outbox.

export function matchingPartitionForPair(
  pairId: number,
  partitionCount = matchingPartitionCount(),
): number {
  return pairId % partitionCount;
}

// pairId -> partition -> queue matching:commands.pN
//                     -> keys  matching:{matching-pN}:*
//                     -> stream matching:{matching-pN}:events

Partition count is topology metadata, not a tuning knob. If producers use 16 and consumers use 32, the same pair reaches two owners and you are back to the lock bug with extra infrastructure. Ours is checked at boot, and a conductor refuses any command whose declared partition is not its own. Keep more logical partitions than physical machines so you can place capacity flexibly, but treat changing the count as a migration: stop routing, drain, snapshot or replay, verify, cut over.

A single hot symbol is a different problem, and the usual ideas do not survive contact with it. Splitting a continuous book by price range fails when an order crosses ranges. Splitting bids from asks fails because every match touches both. Splitting by account fails because price priority spans accounts. Specialised venues do use batch auctions, hardware pipelines, and deterministic parallel algorithms — but those are different execution models with explicit merge rules, not a threading change. For an ordinary continuous limit order book, optimise the single-writer loop and its round trips first.

• Parallelise independent symbols or fixed logical partitions.
• Use bounded queues so downstream pressure reaches admission control instead of memory.
• Keep routing deterministic and reject commands that arrive at the wrong owner.
• Drain, snapshot, replay, and cut over when relocating a live partition.
• Give a dominant symbol dedicated resources long before you consider changing its semantics.

Worker threads run JavaScript in parallel and are genuinely useful for CPU-bound work — independent books, replay verification, compression, computational risk models. They do not make asynchronous datastore or broker calls faster, and spawning a worker per order costs far more than the match itself. If you use them, use a fixed pool or long-lived partition workers with compact commands over bounded channels.

SharedArrayBuffer avoids copies at the price of importing a shared-memory concurrency model into financial state: atomic layouts, publication barriers, ownership transfer, crash handling, version compatibility. For a TypeScript system, separate worker ownership with message passing is far easier to audit. Reach for transferable buffers or a narrow native component when profiling actually shows serialisation is the cost. Do not start with shared mutable arrays because the API exists — the fastest design your team cannot replay deterministically is operationally slow.

This is the part that would have saved us the most time if we had measured it first. We separated the engine boundary from the orchestration boundary and ran each independently:

Decimal arithmetic and the matching function are roughly three orders of magnitude away from being the limit. Redis round trips and confirmed outbox delivery dominate the command path. Adding shared-write threads to the match loop would have complicated ordering while leaving the actual bottleneck completely untouched.

So the higher-leverage work was elsewhere: fixed market partitions, one active conductor per partition, atomic state-and-event commits, and outboxes that publish in parallel across cells with a bounded confirm window. Performance engineering starts by measuring the boundary, not by assuming the serial-looking function is the slow one.

A design is partly defined by what it rules out. These are written down so nobody has to relitigate them under deadline pressure:

• No active-active writers for one pair. Not with locks, not with optimistic retry, not with “it’s fine, the window is tiny.”
• No cross-cell transactions. Settlement across accounts is downstream work driven by a committed trade event.
• No Redlock as an order-book correctness mechanism. It is the thing we removed.
• No changing partition count in place. That is a migration with a drain and a cutover.
• No calling a replica “durable” without an acknowledgement contract behind it.

Feed both implementations the same deterministic command trace: passive limits, orders crossing one and many levels, partial fills, cancels near and far from the touch, duplicate commands, repeated order identities, and bursts concentrated at one best price. Then compare the complete event sequence and the final book — not the elapsed time. If two runs produce different makers, prices, quantities, or ordering from identical input, the faster one is not an implementation of the same market.

Measure p50 through p99.9 alongside sustained throughput, queue age, conflicts, retries, allocation, and CPU. Run long enough to see garbage collection, snapshotting, log rotation, and backpressure. Then start killing things: worker death, network delay, a broker restart mid-burst.

A parallel design earns its complexity only if it raises sustainable end-to-end capacity on representative traffic, produces byte-equivalent outcomes, and stays recoverable. Agree that success criterion before anyone writes the second matching thread — afterwards, sunk cost does the arguing.