Distributed Matching Engine Architecture with Redis and Valkey

Redis and Valkey give you fast data structures, atomic transactions, replication, persistence, Sentinel discovery, and cluster partitioning. Not one of those features decides which order trades first. If two conductors read and mutate the same book concurrently, moving the hashes into a cluster distributes the race rather than resolving it.

The scaling mistake is predictable: add a datastore cluster, run more engine replicas, expect throughput. The replicas then need a distributed lock or optimistic retry around every command, and failover can briefly produce two owners for one pair. You have bought infrastructure and sold correctness.

A matching cell is a simpler unit. A stable set of markets maps to one command queue, one active conductor, one primary/replica storage group, and one outbox. Add cells for independent traffic. The datastore is that cell’s atomic state and event boundary — never a substitute for execution semantics.

Producers compute the partition from the pair and publish to that partition’s durable queue. The active conductor consumes in order, reads candidate makers, stages book changes and events, commits them atomically, waits for the configured durability acknowledgement, and only then acknowledges the command to the broker. A cell-local outbox publishes committed events to settlement and market-data consumers under stable event identities.

pairId -> partition -> RabbitMQ quorum queue (single active consumer)
                              |
                          conductor            (one writer, in order)
                              |
             atomic same-hash-tag MULTI/EXEC
             state + book + dedup + outbox     (all or nothing)
                              |
                Valkey primary --AOF--> disk
                     | WAITAOF 1 1             (then, and only then, ACK)
                     v
                Valkey replica --AOF--> disk
                              |
                   partition outbox
                              |
             bounded publisher confirms
                              |
          at-least-once event + consumer inbox

Partition count is topology metadata, not an ordinary environment variable. If producers use 16 while consumers use 32, the same pair reaches two owners. Choose enough logical partitions for future placement and keep the mapping stable. Several logical partitions can share a physical machine, and a hot one can later get a host of its own. Relocation is a migration: stop routing, drain the queue, snapshot or replay state, verify checksums, change ownership, resume, and watch sequence continuity.

• One deterministic command route per pair.
• One active writer per logical partition.
• One same-slot atomic transaction for state, identity, and outbox.
• One explicit durability policy, applied before the broker acknowledgement.
• One relocation procedure that cannot produce overlapping ownership.

Valkey Cluster divides keys across 16,384 hash slots, and multi-key operations only work when every participating key lands in the same slot. Hash tags force that: only the substring inside braces is hashed. Putting the partition tag in every key of a partition keeps one command transaction local to one primary.

matching:{matching-p3}:order:<orderId>              # HASH  order record
matching:{matching-p3}:book:<pairId>:ask:rates     # ZSET  price index
matching:{matching-p3}:book:<pairId>:bid:rates     # ZSET  price index
matching:{matching-p3}:processed-command:<cmdId>   # STRING command dedup
matching:{matching-p3}:accepted-order:<orderId>    # STRING permanent marker
matching:{matching-p3}:events                      # STREAM outbox
matching:{matching-p3}:events:dead-letter          # STREAM quarantine

The transaction must include every fact whose separation would leave an impossible state. Closing an order removes it from both the order hash and the price index in the same commit. A partial fill updates the remaining quantity and appends its event together. The command dedup record and the accepted-order marker go in too — otherwise a crash between “I matched” and “I remembered that I matched” lets the same order match again.

Cross-partition transactions should be structurally impossible, not merely discouraged. Settlement across accounts is downstream work driven by the committed trade event; it is never a reason to join two books into one storage transaction.

Broker redelivery and client retries look similar and need different defences. A processed-command record handles the same command ID arriving twice after a crash or a lost acknowledgement: the conductor returns the prior result without opening a second state transition.

That does not cover the client that retries the same logical order under a freshly generated command ID. If the first attempt already filled and its live record was deleted, command idempotency has nothing left to recognise. So there is a second marker, keyed by order ID, checked before matching and committed with the first transition — and deliberately never deleted when the order closes.

const acceptedByCommand =
  await this.ordersStorage.getAcceptedOrderCommand(input.id, partition);

if (acceptedByCommand) {
  this.duplicateOrders += 1;            // surfaced as a metric, not a log line
  this.logger.warn('Skipping duplicate matching order identity', {
    orderId: input.id, pairId: input.pairId, commandId, acceptedByCommand,
  });
  return;
}
this.ordersStorage.markOrderAccepted(input.id, commandId, partition, transaction);

Two operational caveats come with this. Enabling it on an existing dataset requires backfilling a marker for every historical and live order — a partial backfill leaves old closed orders replayable. And the marker is deliberately partition-local, so the authoritative orders database still has to enforce globally unique order IDs; making the matching layer reject reuse across partitions would mean adding exactly the global coordination bottleneck the partitioning exists to avoid.

Outbox delivery remains at-least-once regardless. Every financial consumer needs a transactional inbox keyed by the stable event ID, committed in the same database transaction as the projection it drives.

Asynchronous replication can lose an acknowledged write if the primary dies before the replica receives it. The Valkey Cluster specification says this plainly: write safety is best-effort, and acknowledged writes can be lost in some partitions. A financial state service therefore has to define what “success” means rather than inheriting a default.

async ensureDurability(): Promise<void> {
  const { localAofFsyncs, replicaAofFsyncs, timeoutMs } = this.durability;
  if (localAofFsyncs === 0 && replicaAofFsyncs === 0) return;

  const result = await this.redis.call(
    'WAITAOF', localAofFsyncs, replicaAofFsyncs, timeoutMs,
  );

  if (!Array.isArray(result) ||
      Number(result[0]) < localAofFsyncs ||
      Number(result[1]) < replicaAofFsyncs) {
    throw new Error(
      `Matching durability acknowledgement failed: ${JSON.stringify(result)}`,
    );
  }
}

This adds latency by design — the numbers below quantify how much. When the datastore cannot satisfy the contract, the service fails closed rather than quietly degrading to memory semantics. Durability timeouts, replica health, fsync latency, and broker acknowledgement age all become first-class service indicators, because they are now the difference between an acknowledged trade and a lost one.

Memory mode remains available and is a legitimate choice — but only as an explicit declaration that recovery comes from a separate durable log. The startup capability check enforces the distinction: a datastore without WAITAOF support cannot boot in replicated-AOF mode. A persistence checkbox with no tested recovery point objective behind it is not a durability design.

Sentinel and Cluster solve different problems. Sentinel discovers and promotes a primary inside one replicated group. Cluster shards a keyspace across many primaries and redirects clients by hash slot. Because the application already partitions markets, matching cells can start with Sentinel per partition — failure domains and multi-key transactions stay obvious. A larger deployment can place several partition tags in a real cluster, provided the client is genuinely cluster-aware and all transaction keys share a slot.

“Genuinely” is doing work in that sentence. We had a configuration that listed cluster nodes for a client library that constructs a standalone connection. The list was silently ignored. Everything worked in development, and the topology existed only in the config file. That now fails closed:

export function matchingRedisOptions(config: RedisConfig): RedisOptions {
  if (config.REDIS_CLUSTER_NODES) {
    throw new Error(
      'REDIS_CLUSTER_NODES requires a real ioredis Cluster client; ' +
      'use partitioned matching cells or wire ClusterModule explicitly',
    );
  }
  // ... Sentinel discovery when REDIS_SENTINEL_HOSTS and _NAME are both set
}

Capability checks should prove the instantiated client mode, the Sentinel master name, persistence command support, and the assigned partition — at boot, in the process, against the real connection. A conductor also rejects any command whose declared partition is not its own. Failover testing then means stopping a primary, watching promotion, verifying the durability boundary held, and comparing book and event state before letting traffic back in.

Publishing to RabbitMQ after committing to Redis creates a dual-write gap: the engine can crash after the state changed but before the event was sent, and the trade exists in the book but never reaches settlement. Appending the event to a Redis stream inside the state transaction closes that gap. A separate worker reads the stream in order and publishes with confirms.

The remaining gap is smaller and honest: a crash after a broker confirm but before acknowledging the stream entry republishes the event. Stable event IDs make that detectable, which is why every consumer needs an inbox. We do not claim exactly-once delivery across Redis and RabbitMQ, because it is not achievable and claiming it just means the duplicate handling lives somewhere nobody has looked.

Sequential publish-and-confirm preserves order but caps a worker at a few hundred events per second — which is precisely what our measurements showed. A bounded confirm window fixes the throughput without losing the ordering:

// Publish without awaiting each result: a bounded window of confirms
// stays open, and RabbitMQ receives the calls on one channel in stream order.
const confirmations = deliveries.map(({ entry, event }) =>
  publishEvent(rabbit, entry, event),
);
const confirmed = await Promise.allSettled(confirmations);

if (confirmed.some((r) => r.status === 'rejected' || !r.value)) {
  throw new Error('RabbitMQ did not confirm publication batch');
}

// Only now advance the stream, as one transaction.
const transaction = dataClient.multi();
transaction.xack(streamKey, consumerGroup, ...streamIds);
transaction.xdel(streamKey, ...streamIds);
transaction.hdel(attemptsKey, ...eventIds);

Note the allSettled rather than all: if any confirm fails you still wait for every outstanding promise to settle before retrying or closing the channel, or you leave callbacks firing against a channel you already tore down. And watch queue age and the oldest unacknowledged stream entry rather than raw backlog count — the count tells you how much is waiting, the age tells you how long your financial projections have been wrong.

The synthetic workload writes a hash, a sorted-set member, and a stream event in one transaction — the conductor’s access pattern in miniature. We ran it against one cell and two cells to test the horizontal scaling claim.

This disconfirms the naive version of our own architecture claim, which is exactly why it is worth publishing. Cells scale when traffic is independent and resources are isolated. A second cell on the same constrained host is two processes fighting over one CPU scheduler, one memory bus, and one disk.

It also puts the durability cost in perspective: replicated AOF acknowledgement is roughly an 8x throughput reduction against memory mode on the same hardware. That is the price of the recovery point objective, stated honestly rather than buried.

The next meaningful test places cells on separate hosts and requires at least 1.5x aggregate throughput going from one cell to two, with zero invariant violations under sustained load and during failover. Until that passes, the multi-cell design is a hypothesis with a plausible mechanism, not a demonstrated result.

Observe every partition independently, and keep the partition dimension on every metric. A healthy aggregate hides one stalled market perfectly. Track command age, processing percentiles, wrong-partition rejects, duplicate commands, duplicate order identities, event creation rate, datastore latency, AOF acknowledgement latency, replication offset, outbox age, confirm failures, and consumer inbox duplicates.

The engine also exposes an on-demand consistency check, because the invariant “every indexed order exists and matches its index” is cheap to verify and expensive to violate silently:

{
  "pairId": 12, "partition": 3,
  "indexedOrders": 8412, "scannedOrders": 8412,
  "truncated": false, "healthy": false,
  "issues": [
    "ask:7f3a…:missing-order",   // in the index, no order record
    "bid:91c2…:index-mismatch"   // record disagrees with its index
  ]
}

Capacity planning follows the busiest symbol, not total traffic divided by worker count. Reserve headroom for bursts, snapshot work, failover replay, and a temporarily absent replica. Put a hot partition on dedicated resources long before you consider changing its execution model — and if one market genuinely reaches its single-writer ceiling after data-layout and round-trip work, that is a market-policy or hardware decision to make explicitly, not a threading change to sneak in.

Distributed Redis is most valuable when it keeps independent state independent. It is least useful when it is used to disguise shared ownership.