WebSocket Architecture for Real-Time Market Data: Following One Trade Through an Exchange Gateway

My portfolio says I built a WebSocket-based trading system that delivers instant price updates to thousands of concurrent users. This article opens up the part of that system that faces the browser: a service called users-ws in the Bitsten backend, which is a NestJS monorepo. I will walk through it with the real code, including the parts I would not ship again.

Some terms first. A WebSocket (RFC 6455) is a long-lived, two-way connection that starts life as an HTTP request and is then upgraded, so the server can push bytes whenever it wants. Socket.IO is a library on top of that. It adds named events, automatic reconnection on the client, and rooms: named groups of sockets on the server that you can broadcast to with one call. Fan-out is the step where one internal event becomes many socket writes.

An exchange gateway carries two very different kinds of traffic. Public market data is the order book, the trade tape, candles, and the 24-hour ticker for each trading pair. Every subscriber gets the same bytes. Private user data is one user's orders, fills (Bitsten calls a fill a deal), balances, and futures positions. It must reach every tab that user has open and nobody else.

In one sentence, users-ws is a single Node.js process that consumes RabbitMQ messages from other services, calls those services back over RabbitMQ RPC when a message does not carry enough detail, and writes the result to Socket.IO sockets. In production it ran as one replica, and the scaling section explains why.

The service has three feature modules: spot, futures, and shared (cross-product balances). Each module has the same three parts. A controller holds the RabbitMQ consumers. A gateway holds the Socket.IO handlers and the code that emits to sockets. A service wraps RPC calls to other backend services through small proxy classes.

exchanges: [
  ...Object.values(SpotExchanges),
  ...Object.values(SharedExchanges),
  ...Object.values(FuturesExchanges),
].map((exchange) => ({
  name: exchange,
  type: 'direct',
})),
...
deserializer: (message: Buffer) => {
  const decodedMessage = SuperJSON.parse(message.toString());
  return decodedMessage;
},

The browser talks to the gateway with two events, subscribe and unsubscribe, each carrying a room name. Public room names carry a pair id. Private rooms such as orders and balances carry nothing that identifies the user, which becomes important in the next section.

export const SPOT_ROOMS = {
  tradesByPair: (pairId: number) => `trades:${pairId}`,
  balances: () => 'balances',
  orderBookByPair: (pairId: number) => `order_book:${pairId}`,
  chartsByPair: (pairId: number, interval: string) =>
    `charts:${pairId}:${interval}`,
  pairsState: () => 'pairs_state',
  dealsByPair: (pairId: number) => `deals:${pairId}`,
  deals: () => `deals`,
  ordersByPair: (pairId: number) => `orders:${pairId}`,
  orders: () => `orders`,
};

All three gateways extend one base class and share one Socket.IO server on one port. The web client uses a single connection for all rooms, so every subscribe event reaches all three gateways. Each one tests the room name against its own regex table and ignores names it does not own.

Opening a connection needs no credentials, and anyone can join a public room. Authentication happens when a socket asks for a private room. A guard reads the Access cookie from the handshake headers (the browser sends cookies on the WebSocket upgrade request) and asks the users service to verify it over RPC.

if (!data.room) return true;
if (!options.protectedRooms.length) return true;
if (!options.protectedRooms.find((regexp) => regexp.test(data.room)))
  return true;

try {
  const token =
    client?.handshake?.headers?.cookie &&
    cooker.parse(client.handshake.headers.cookie)['Access'];
  ...
  const userId = await this.usersProxy.verifyJwtToken({
    token: token,
  });
  ...
  client.user = { id: userId };

  return true;
} catch (err) {
  client.emit('error', err?.message || err);
  return false;
}

const userId = +(await this.service.verifyAsync(payload.token)).sub;
const isLastTokenRevoked = await this.jwtStorage.isLastTokenRevoked(
  userId,
);
if (isLastTokenRevoked) return null;
return userId;

Once the guard passes, the base gateway joins the socket to the room and records it in an in-memory map from user id to that user's sockets. Private delivery goes through this map, never through a room broadcast:

handlePrivateJoin(room: string, socket: SocketWithUser) {
  socket.join(room);
  const userId = socket.user.id;
  const userClients = this.usersClients.get(userId);

  this.usersClients.set(
    userId,
    userClients
      ? Object.assign(userClients, { [socket.id]: socket })
      : { [socket.id]: socket },
  );
  ...
}

public castOrder(order: Order) {
  const clientsByUser = this.usersClients.get(order.userId);
  ...
  for (const client of Object.values(clientsByUser)) {
    if (client.rooms.has(ordersByPairRoom))
      client.emit(ordersByPairRoom, { orders: [order] });
    if (client.rooms.has(ordersRoom))
      client.emit(ordersRoom, { orders: [order] });
  }
}

So every logged-in user who watches their orders is a member of one shared room literally named orders. The room works as a flag on the socket ("this tab wants order updates"), and the identity check comes from the map. No code path broadcasts to that room, but one careless this.server.to('orders').emit(...) would send one user's order to every logged-in user, and nothing in the design stops it. I would name private rooms per user, as in user:42:orders, so that a broadcast mistake stays inside one account. Per-user room names are also what a Socket.IO cluster adapter needs.

Two more gaps. First, the token is checked once, when the room is joined. If the user logs out in another tab, the revocation flag is set, but the open socket keeps receiving private data until it disconnects. The fix is to store the token expiry on the socket, disconnect when it passes, and have the users service publish a revocation event that the gateway turns into disconnect() for that user's sockets. Second, the gateway sets cors: { origin: process.env.ORIGIN_URL }. Browsers do not apply CORS to WebSocket connections, and the web client uses the WebSocket transport only, so that option does not stop another site from opening a socket. What stops a hostile page from using the victim's session is the cookie itself: the users service sets Access with sameSite: 'lax', so browsers leave it off cross-site WebSocket handshakes. I would still check the Origin header explicitly in Socket.IO's allowRequest hook.

Trader A has a resting limit sell on a pair. Trader B sends a market buy that fills it. Both have the exchange open, and other people are watching that pair's order book and trade tape. These are the hops, in order:

1. The conductor (the service that runs matching) writes a deal.executed.v1 event and the order events for both orders into a Redis stream in the same transaction as the book change. This is a transactional outbox.
2. The conductor-outbox service reads the stream, publishes each event to RabbitMQ with publisher confirms, and only then acknowledges and deletes the stream entry.
3. RabbitMQ copies the messages:deal:create message into every queue bound to that exchange: one for users-ws, and others for aggregates, the calculator, and bots.
4. users-ws turns the deal into a public trade and broadcasts it to the trades:<pairId> room. The trade tape is updated.
5. In parallel, the calculator computes fees and publishes messages:aggregates:create-user-deal-with-fee. Aggregates stores each side's deal, commits, and publishes messages:aggregates:deal-created with both user ids and both order ids.
6. users-ws receives that message and, for each trader, fetches the deal and the order from the users service over RPC, then pushes them to that user's sockets through the map shown above.
7. Separately, the users service stores each order change and publishes messages:users:order-updated. users-ws fetches the order again and pushes it again. The client merges by order id, so the duplicate is harmless.
8. The balances service publishes messages:balances:balance-updated for each balance increase. Two users-ws queues receive it (spot and shared), and they make one and four RPC calls respectively to compute the balance views they push.
9. None of this updates the order book. It changes at the next one-second polling tick.

return rabbit.publish(event.exchange, event.routingKey, payload, {
  persistent: true,
  mandatory: true,
  messageId: event.eventId,
  correlationId: event.correlationId,
  type: event.type,
  headers: {
    'x-event-id': event.eventId,
    'x-schema-version': event.schemaVersion,
    'x-causation-id': event.causationId,
    'x-partition': event.partition,
    'x-partition-offset': entry.streamId,
  },
});

@RabbitSubscribe({
  exchange: SpotExchanges.handleDeal,
  routingKey: SpotExchanges.handleDeal,
  queue: SpotExchanges.handleDeal + ':users-ws-spot',
})
async handleDeal(payload: ConductorDeal) {
  const pair = await this.assetsProxy.getPairBy({
    fieldName: 'id',
    value: payload.taker.pairId,
  });
  ...
  const room = SPOT_ROOMS.tradesByPair(payload.taker.pairId);
  this.gateway.castTrades(room, [
    {
      id: payload.id,
      price: new Decimal(payload.takerQuoteVolumeChange)
        .div(payload.takerVolumeChange)
        .toString(),
      ...
      isBuy: payload.taker.side === 'bid',
    },
  ]);
}

castTrades ends in this.server.to(room).emit(room, trades). Socket.IO's in-memory adapter encodes the packet once and writes the same encoded bytes to every socket in the room, so a busy public room costs one JSON encode plus one write per subscriber.

async handleDealCreated(payload: RawDeal) {
  if (payload.makerUserId) {
    const deal = await this.service.getDeal({
      userId: payload.makerUserId,
      id: payload.id,
    });
    if (deal) this.gateway.castDeal(deal);
    if (payload.makerOrderId) {
      const order = await this.service.getOrder({
        userId: payload.makerUserId,
        id: payload.makerOrderId,
      });
      if (order) this.gateway.castOrder(order);
    }
  }

  if (payload.takerUserId) {
    ...
  }
}

Count the RPC calls users-ws makes for this single fill: one for the public trade, four in handleDealCreated (deal and order for each side), one per order-updated message (two orders, so two), and five per balance-updated message, of which there are at least two because both traders receive something. That is at least seventeen RPC round trips to the assets, users, and balances services, all made after the matching engine has already finished. This pattern is called fetch-on-notify: the event says "something changed", and the consumer asks for the current state. Messages stay small, but the gateway's cost scales with events times RPC latency, and every RPC target sits on the critical path of a push.

One more thing in this path. The matching engine publishes through an outbox, but aggregates publishes deal-created with a plain this.rabbit.publish(...) after its database commit. If the process dies between the commit and the publish, the private deal push for both traders is lost. They see the fill the next time the page fetches deals over REST. For a UI notification that is acceptable, and I would document it next to the publish call.

The order book does not use events at all. Every second, the controller asks the assets service for all tradable pairs, fetches each pair's book from aggregates, and broadcasts a full snapshot to that pair's room. A new subscriber gets an immediate snapshot sent only to its own socket, so it does not wait for the next tick.

@Interval(1000)
async castOrderbook(): Promise<void> {
  const pairs = await this.assetsProxy.getAvailableToTradePairs();

  await Promise.all(
    pairs.map(async (pair) => {
      const room = SPOT_ROOMS.orderBookByPair(pair.id);
      const orderbook = await this.service.getOrderbook(pair.id);
      this.gateway.castOrderbook(room, orderbook);
    }),
  );
}

private async getAsks(exchangePairId: number): Promise<Order[]> {
  const key = `matching:orderBook:${exchangePairId}:ask:rates`;
  const data = await this.redis.zrangebyscore(
    key,
    '-inf',
    'inf',
    'WITHSCORES',
  );
  ...
  for (let i = 0; i < data.length; i += 2) {
    const targetKey = `orders:${data[i]}`;
    const dto = await this.redis.hgetall(targetKey);
    ...

After reading both sides, aggregates groups orders into price levels at the pair's first precision setting and keeps the top 50 levels (the RPC's default limit). The browser hook replaces its whole order book state with each message.

This is conflation: replacing a queue of updates with only the latest value. Whether the book changed once or a thousand times in a second, each subscriber gets one message. There are no deltas, so there is nothing to sequence. A lost message is repaired by the next one a second later. A client that reconnects is correct within a second without any resync protocol.

The costs, each visible in the code:

• Latency floor. The trade tape is pushed per event, but the book arrives up to a second later. A user can see a trade at a price while the book still shows that level.
• Bandwidth. Every subscriber gets 50 levels per side every second, even when nothing changed.
• Snapshot consistency. Asks and bids are read in separate Redis calls, and each order hash is its own HGETALL, while the matching engine keeps writing. A snapshot is not taken at one point in time, so in principle it can show a crossed book for one tick.
• Overlapping ticks. @Interval uses setInterval and does not wait for the previous async run. The RPC timeout in the RabbitMQ library defaults to 10 seconds, so a slow tick can overlap the next one, and the older snapshot for a pair can arrive after the newer one.
• Work without subscribers. Every tradable pair is fetched every second whether anyone watches it or not. Emitting to an empty room is free, but the RPC calls and Redis reads are not.

Candles use a smaller version of the same idea. Aggregates publishes kindle-updated on every change. The gateway keeps only the latest candle per pair and interval in a map and flushes that map once per second. The 24-hour ticker (pairs_state) is also recomputed and broadcast every second. Trades are the only public stream that is not conflated.

async handleKindleUpdated(payload: { ... }) {
  const { exchangePairId, interval } = payload;

  if (this.kindlesUpdate[exchangePairId]) {
    this.kindlesUpdate[exchangePairId][interval] = payload;
  } else {
    this.kindlesUpdate[exchangePairId] = { [interval]: payload };
  }
}

The textbook design for a real-time order book over WebSocket is a snapshot plus deltas. Each delta carries a sequence number. The client buffers deltas, fetches a snapshot that says which sequence number it reflects, drops deltas older than that, and applies the rest. If it ever sees a gap in the numbers, it throws its book away and starts over. Binance documents this procedure for its public streams.

Bitsten's gateway has no sequence numbers anywhere in what it sends. The interesting part is that the information exists one hop upstream. The outbox publisher puts the Redis stream entry id in the x-partition-offset header, and stream ids only increase within a partition. The conductor also writes an aggregateVersion into every event envelope. The RabbitMQ library passes the raw AMQP message and its headers to a handler as extra arguments, but none of the users-ws handlers declare them, so the ordering information is dropped at the gateway's front door.

For the order book this has not mattered, because full snapshots once a second are their own gap recovery. For private updates it matters more. Fetch-on-notify means each push carries the full current state of one order, and the client upserts by id. Two notifications for the same order can be processed at the same time (the consumer allows ten unacknowledged messages in flight), and their RPC responses can come back in either order. If the fetch that started first finishes last, the client shows "partially filled" after "filled" until something else changes that order.

What I would change, in order of effort:

1. Carry a per-order version on every private push and have the client ignore anything older than what it already shows. The users service already stores the order row, so a version column there is cheap. This fixes the reordering without touching the transport.
2. For the book, have the snapshot RPC return the stream offset it was read at, forward x-partition-offset on deltas, and give the client the standard buffer, snapshot, discard, apply, resync-on-gap loop. Keep the one-second snapshot as a fallback for clients that fall behind.
3. Read both sides of the book in one Redis MULTI or one Lua script so a snapshot is internally consistent.

Backpressure means a slow reader forces the writer to slow down or drop data, instead of letting a buffer between them grow without limit. The gateway sits between two such buffers: the RabbitMQ queue in front of it and one send buffer per socket behind it.

On the RabbitMQ side, the @golevelup/nestjs-rabbitmq defaults apply: a prefetch of 10, and failed handlers are requeued. Prefetch is the number of unacknowledged messages RabbitMQ will hand a consumer at once, so it caps concurrency, and when an RPC target is slow, the backlog waits in RabbitMQ. The requeue default is the weak part. If the taker-side RPC in handleDealCreated times out, the whole message goes back to the queue, and the maker gets the same push again on redelivery. A message that always fails is redelivered forever. I would reject to a dead-letter queue after a few attempts.

On the socket side there is no backpressure at all. Here is where a broadcast ends up in Socket.IO:

writeToEngine(encodedPackets, opts) {
    if (opts.volatile && !this.conn.transport.writable) {
        debug("volatile packet is discarded since the transport is not currently writable");
        return;
    }
    ...
    for (const encodedPacket of packets) {
        this.conn.write(encodedPacket, opts);
    }
}

conn.write pushes onto engine.io's per-socket writeBuffer, a plain array with no size limit, which is flushed whenever the transport can take more. A phone on a bad network subscribed to a few order books keeps accumulating one snapshot per book per second in the gateway's memory until the heartbeat declares it dead (engine.io defaults: a ping every 25 seconds and 20 seconds to answer). The gateway never uses the volatile flag.

The fix depends on the stream. Conflated streams (order book, ticker, candles) should be sent with this.server.to(room).volatile.emit(...). Dropping a snapshot for a socket that cannot keep up is correct, because the next one replaces it. Private streams should never be dropped silently, so for those I would check socket.conn.writeBuffer.length before emitting and, above a threshold, disconnect the socket with a reason. The client reconnects and refetches its orders over REST, which is a clean resync. I would also cap how many rooms one socket can join, since today a single connection can subscribe to every pair's book.

A reconnect storm is what happens when many clients lose their connection at the same moment (a deploy, a pod restart, a load balancer change) and all come back at once, each re-subscribing and asking for initial state. Here is the client side:

this.socketClient = io(process.env.NEXT_PUBLIC_SOCKET_URL, {
  path: '/socket.io',
  timeout: 2000,
  transports: ['websocket'],
})
...
private setupSocket() {
  this.socketClient.on('disconnect', this.retrySubscription.bind(this))
  this.socketClient.on('connect', () => {
    this.retryDelay = this.MIN_DELAY
    this.subscriptions.forEach((callback, eventName) => {
      this.socketClient.emit('subscribe', { room: eventName })
      this.socketClient.on(eventName, callback)
    })
  })
}

The actual reconnecting is done by Socket.IO's built-in manager with its defaults: first retry after one second, growing to at most five, with a randomization factor of 0.5. That random jitter is what spreads a storm out, and it is on by default. The custom retrySubscription method only schedules timers that check connected and double a delay; it never calls connect(), so its 60-second cap has no effect.

The connect handler has a real bug. It registers socketClient.on(eventName, callback) again on every connect, and Socket.IO client listeners survive reconnects. A tab that has reconnected three times runs each callback four times per message. The order book replaces its state and the trade tape is keyed by trade id, so both hide it. The orders hook calls notifications.showOrderNotification inside its merge, so that one would repeat its notification once per extra listener. The fix is to register listeners once and only re-emit subscribe on connect.

On the server, each re-subscription costs RPC calls: an order book or trade tape subscription fetches its initial snapshot through two RPCs, and each private room runs the JWT check through another. With one replica, a deploy disconnects everyone at once. Three changes would make that cheaper. Serve initial order book snapshots from the snapshot the gateway already builds every second, instead of a fresh RPC. Verify the token once per connection instead of once per private room. Add a connection rate limit in allowRequest so a storm queues at the edge instead of at the users service.

One infrastructure detail: the ingress is a GCE load balancer, where the backend service timeout also caps how long a WebSocket may stay open, with a default of 30 seconds. The repository's charts set no BackendConfig timeout for users-ws. I would set it explicitly next to the service.

deployment:
  replicas: 1
...
deployment:
  command: npm
  args:
    - run
    - start:users-ws
  resources:
    ...
    limits:
      cpu: 900m
      memory: 900Mi

Setting replicas: 2 would break the gateway without any error. Look at the queue names in the controllers: messages:deal:create:users-ws-spot, messages:users:order-updated, messages:futures:trades. They are fixed strings. Two replicas would share those queues, and RabbitMQ delivers each message in a queue to only one consumer (the competing consumers pattern). Each replica would get about half the trades and emit them only to its own sockets, so every user would see about half the tape. Private pushes would fail the same way, and even worse, since usersClients lives in each process's memory: the replica that receives trader A's deal may not be the one holding trader A's socket.

Sticky sessions (the load balancer pinning a client to one backend) are not needed today. Socket.IO needs them when a client uses the HTTP long-polling transport, because each poll is a separate HTTP request that must reach the process holding that session. This client sets transports: ['websocket'], so the whole session is one upgraded TCP connection and it stays on whichever pod accepted it. The price is losing users behind proxies that block WebSockets.

There are three ways to make more than one replica correct. The table below compares them. What I would build is a mix of the first and third. Public market data goes to a per-replica queue, because every replica needs every public event anyway. Private events get their enrichment moved upstream so the message already carries the full order or deal, then get published to a topic exchange with a routing key like user.42. Each replica binds that key when a user's first socket connects and unbinds it when the last one leaves. A replica then receives only the private events for users it holds, and no replica makes an RPC call per event. The cost is binding churn on RabbitMQ when many users connect at once.

Besides the client listener bug, the one-time auth check, and the overlapping timers covered above, these are the defects I can point at in the code. Most were hidden by the client merging by id.

@RabbitSubscribe({
  exchange: SpotExchanges.handleBalanceUpdated,
  routingKey: SpotExchanges.handleBalanceUpdated,
  queue: SpotExchanges.handleBalanceUpdated + +':users-ws-spot',
})

• A queue named NaN. +':users-ws-spot' is a unary plus applied to a string, which yields NaN, so the queue is actually called messages:balances:balance-updatedNaN. It still works, because the queue is bound to the right exchange. But the service map documents balance-updated:users-ws-spot, and anyone looking for that queue in the RabbitMQ console finds nothing.
• Futures subscriptions are not reference counted. Joining a futures private room asks the futures service to subscribe the user's account to the upstream topic, and leaving asks it to unsubscribe. The futures service keeps one upstream subscription per user and topic. With two tabs open, leaving the page in one tab unsubscribes the user for both, and a disconnect without an explicit unsubscribe never unsubscribes at all. The gateway should count sockets per user and topic, and only call upstream on the first join and the last leave, including leaves caused by disconnect.
• The user map never shrinks. handleDisconnect deletes the socket id from a user's entry but never deletes the entry, so the map keeps one empty object per user who has ever connected since the process started.
• Five RPC calls per balance message. The balances service publishes only { userId }, and users-ws asks for spot, aggregated, available, staking, and total balances separately. Publishing the new balance in the event would remove all five.

The larger change is about where work happens. The gateway knows how to compute decimals, which balance views exist, and how to build a trade from a deal, which is why it makes so many RPC calls. I would make upstream services publish complete, versioned payloads, already addressed to a user or a room, and keep the gateway to four jobs: authenticate, track subscriptions, apply backpressure, and write bytes. A gateway like that can be scaled by adding pods, and a slow users or balances service no longer delays pushes.

For a retail web interface with a modest number of pairs, one replica with one-second snapshots is a reasonable design. These are its limits, in the order I expect them to be hit:

• Private event throughput. handleDealCreated makes four sequential RPC calls with at most ten messages in flight, so its throughput is roughly ten divided by the time of four RPC round trips. The matching engine handles 4,000+ orders per second. If a meaningful share of those produce fills between logged-in users, this consumer is the bottleneck well before Socket.IO is.
• Order book read cost. Each tick reads every resting order on both sides of every pair, one HGETALL per order, before grouping and cutting to 50 levels. The cost grows with total book depth and is paid every second, whoever is watching.
• One event loop. All sockets, all RPC handling, and all JSON encoding share one Node.js process capped at 900m CPU. Room broadcasts encode once, but private pushes encode per socket, and a burst of fills competes with the order book tick for the same loop.
• Snapshot bandwidth. Full books once a second per subscriber is fine for a browser, and wasteful for anyone who wants many pairs at once.
• The one-second floor. Market makers and API traders need deltas with sequence numbers. The public tape is push, but the book will never be faster than the polling interval.
• The first extra replica. As shown above, scaling out needs queue and routing changes before it is correct, so there is no quick way to add capacity under load.

If I were starting this service again, I would keep the room model, the websocket-only transport, and conflation for everything that can be conflated. I would add per-user routing, versions on private payloads, volatile emits for snapshots, and sequence numbers on the book from the first day, because adding them later means changing the producer, the gateway, and the client at the same time.