Microservices Communication Patterns: RPC vs Events in a Crypto Exchange Backend

Bitsten is a cryptocurrency exchange, and I was Technical Lead on its backend. The backend is a NestJS monorepo with about seventeen services under apps/. The ones that matter for this article: users (the HTTP API that customers call), users-ws (the WebSocket gateway that pushes updates to browsers), calculator (reserves funds for new orders and settles trades), balances (the ledger), assets (the catalogue of coins, networks, and trading pairs), conductor (the matching engine), conductor-outbox (publishes what the engine committed), and aggregates (order book snapshots, candles, deal history). Around them sit bots, custody, kyc, admin, futures, and a few others.

Almost all traffic between these services goes through one RabbitMQ broker, and the message names tell you which of two styles each one uses. A name that starts with rpc: is request/reply: the caller sends a message and waits for an answer, the same way it would wait on an HTTP call. That is synchronous communication in the sense that matters, because the caller cannot continue until the callee answers, even though a broker sits in the middle. A name that starts with messages: is an event: the producer announces that something happened and moves on. Zero, one, or five services may be listening, and the producer does not know which.

The repository has a hand-drawn map of these calls, service-communication.mmd. While writing this I checked every edge I discuss against the code, and the map is wrong in three places. Someone changed a call from an event to a request in a pull request, and the diagram kept the old arrow.

Below: how request/reply is built on RabbitMQ, one limit order traced hop by hop, where synchronous chains fail, what I would change, and a rule for choosing between a call, an event, and a local copy. The matching engine and its outbox have their own write-up, Distributed Matching Engine Architecture with Redis and Valkey, so here they appear only as two hops in the chain.

NestJS ships its own RabbitMQ transport in @nestjs/microservices, and apps/aggregates/src/main.ts still connects one, to a queue called conductor:queue that looks like a leftover. Everything else uses @golevelup/nestjs-rabbitmq version 5.7.0. It gives each service an injectable AmqpConnection with two methods that matter: publish for events and request for RPC. On the receiving side there are two decorators, @RabbitSubscribe for events and @RabbitRPC for requests. Every service configures SuperJSON as the serializer, so bigint and Date values survive the trip, which matters for a ledger that stores amounts as 18-decimal integers.

A request needs three things that a plain publish does not: a way to route the answer back, a way to match the answer to the question, and a limit on how long to wait. Here is how the library does all three.

    async request(requestOptions) {
        ...
        const correlationId = requestOptions.correlationId || (0, crypto_1.randomUUID)();
        ...
        const timeout = requestOptions.timeout || this.config.defaultRpcTimeout;
        const payload = requestOptions.payload || {};
        const response$ = this.messageSubject.pipe((0, operators_1.filter)((x) => requestId
            ? x.correlationId === correlationId && x.requestId === requestId
            : x.correlationId === correlationId), (0, operators_1.map)((x) => x.message), (0, operators_1.first)());
        const timeout$ = (0, rxjs_1.interval)(timeout).pipe((0, operators_1.first)(), (0, operators_1.map)(() => {
            throw new Error(...);
        }));
        const result = (0, rxjs_1.lastValueFrom)((0, rxjs_1.race)(response$, timeout$));
        await this.publish(requestOptions.exchange, requestOptions.routingKey, payload, Object.assign(Object.assign({}, requestOptions.publishOptions), { replyTo: DIRECT_REPLY_QUEUE, correlationId, headers: requestOptions.headers, expiration: requestOptions.expiration }));
        return result;
    }

• Routing the answer back. DIRECT_REPLY_QUEUE is amq.rabbitmq.reply-to, RabbitMQ's direct reply-to feature. It is a pseudo-queue: there is no real queue behind it. The broker delivers the reply straight to the consumer on the caller's channel. That saves declaring a reply queue per caller, and it also means a reply can only reach the exact channel that sent the request.
• Matching answer to question. Every request gets a random UUID as its correlation id. All replies arriving on the channel go into one RxJS subject, and each pending request filters that stream for its own id.
• Waiting. The response stream races a timer. The default is defaultRpcTimeout: 10000, ten seconds, and no service in the repository changes it. One call site passes its own timeout (the bots service, 5 seconds) and one passes an expiration (a futures order book query in users-ws, 2 seconds).

The last line is the one to read twice. The timeout lives only in the caller's memory. The request message carries an expiration, which is RabbitMQ's per-message time to live, only if the caller passes one, and almost none do. So when a caller gives up after ten seconds, its request is still sitting in the callee's queue. The callee will execute it later and send the reply to a channel that has stopped listening. For a read that wastes work. For a write, the side effect happens after the caller has already reported failure.

The callee side is short. It consumes the request, runs the handler, publishes the return value to replyTo with the same correlation id, and acknowledges. The part that matters is what happens when the handler throws:

const defaultConfig = {
    name: 'default',
    prefetchCount: 10,
    defaultExchangeType: 'topic',
    defaultRpcErrorHandler: (0, errorBehaviors_1.getHandlerForLegacyBehavior)(errorBehaviors_1.MessageHandlerErrorBehavior.REQUEUE),
    defaultSubscribeErrorBehavior: errorBehaviors_1.MessageHandlerErrorBehavior.REQUEUE,
    ...
    defaultRpcTimeout: 10000,
...
            catch (e) {
                ...
                    const errorHandler = rpcOptions.errorHandler ||
                        this.config.defaultRpcErrorHandler ||
                        (0, errorBehaviors_1.getHandlerForLegacyBehavior)(rpcOptions.errorBehavior ||
                            this.config.defaultSubscribeErrorBehavior);
                    await errorHandler(channel, msg, e);

REQUEUE means channel.nack(m, false, true): tell the broker the message failed and put it back. There is no delay and no attempt counter, so the broker hands the same message out again right away. Event subscribers get the same default through defaultSubscribeErrorBehavior. None of the Bitsten services override either default, and I could not find a dead-letter exchange configured on any queue. I come back to what that does in the failure section.

Every endpoint gets its own direct exchange (an exchange that routes a message to queues whose binding key equals the routing key), and the exchange name, routing key, and queue name are the same string. Each service keeps a map from handler name to that string and declares the exchanges at startup.

export const AssetsExchanges: Record<keyof AssetsController, string> = {
  create: 'rpc:assets:create',
  get: 'rpc:assets:get',
  getOneBy: 'rpc:assets:get-one-by',
  ...
};
...
  @RabbitRPC({
    exchange: AssetsExchanges.getOneBy,
    routingKey: AssetsExchanges.getOneBy,
    queue: AssetsExchanges.getOneBy,
  })
  public async getOneBy<
    F extends keyof Pick<Asset, 'id' | 'name' | 'code'>,
    V extends Asset[F],
  >(payload: { fieldName: F; value: V }): Promise<Result<Asset | null>> {
    try {
      const asset = await this.service.getOneBy(
        payload.fieldName,
        payload.value,
      );
      return ok(asset);
    } catch (e) {
      return error(e);
    }
  }

For RPC, one queue per endpoint is what you want. If you run two copies of assets, both consume from rpc:assets:get-one-by, and each request goes to one of them. RabbitMQ calls this competing consumers.

Events need the opposite: every interested service should get its own copy. The convention for that is a suffix on the queue name. When the outbox publishes messages:order:delete, calculator binds a queue called messages:order:delete:calculator and users binds messages:order:delete:users. Two queues on the same exchange, so each service gets every message, and replicas of one service still share their queue. The queue name alone decides whether two processes split the work or each get a copy, and a missing suffix silently turns one into the other. users-ws subscribes to messages:users:order-updated with no suffix, which is harmless with one process and wrong with two. More on that later.

The last convention is the reply shape. Most handlers return Result<T> from libs/shared-lib/src/interfaces/Result.ts, which is either { data } or { error: { message, code } }. Expected business failures travel as values, and the caller checks isError(result). That works when it is applied consistently. It is not: rpc:assets:get returns a bare array and, on a database error, logs it and returns [], so a caller cannot tell an empty catalogue from a failed query. Balances returns NEGATIVE_BALANCE as a value but rethrows every other error, which sends the request back through the requeue path above.

Here is what happens when a logged-in customer places a limit order to sell. I label each hop as RPC or event.

1. The browser sends POST /orders/limit to users. An auth guard and a throttler run first.
2. users to assets, RPC rpc:assets:get-one-pair-by. Users needs the pair's minimum and maximum amounts and tick sizes to validate the order.
3. users to calculator, event messages:calculator:add-order. Users does not wait for it. The HTTP response goes back with the new order id.
4. calculator to assets, RPC, the same pair lookup again.
5. calculator to balances, RPC rpc:balances:increase-decimals with a negative amount. This reserves the funds, and the answer decides whether the order continues.
6. calculator to conductor, RPC on matching:commands.p0, a versioned place-order command. The conductor matches it, then commits the book change and the resulting events to Redis in one transaction.
7. conductor-outbox, events messages:order:create, messages:deal:create, and others, published with broker confirms.
8. users consumes messages:order:create, saves its copy of the order, and publishes another event, messages:users:order-updated, carrying only { id, userId, pairId }.
9. users-ws consumes that and calls back, RPC rpc:users:get-order. To build the reply, users makes two more RPCs: assets for the pair and aggregates for the order's deals. Then users-ws pushes the order to the customer's socket.
10. Separately, balances published messages:balances:balance-updated when the funds were reserved in step 5. users-ws consumes it and asks users to recompute the customer's whole portfolio, which is a long series of further RPCs covered below.

On the path from the click to the order showing up in the browser, ignoring the balance refresh, that is seven request/reply round trips and three events. Here are the first two hops that write something:

  public async createLimitOrder(
    dto: CreateLimitOrderDto,
    userId: number,
  ): Promise<string> {
    const id = uuid();
    ...
    const exchangePair = await this.assetsProxy.getPairBy({
      fieldName: 'id',
      value: dto.exchangePairId,
    });
    if (!exchangePair) return null;

    await this.validateLimitOrderCreateInfo(exchangePair, dto);

    this.calculatorProxy.addOrder({
      id,
      userId,
      type: OrderType.limit,
      ...
    });
    ...
    return id;
  }

// calculator.proxy.ts
  addOrder(payload: { ... }) {
    this.rabbit.publish(
      'messages:calculator:add-order',
      'messages:calculator:add-order',
      payload,
    );
  }

    if (order.type === 'limit' && order.side === 'ask') {
      const result = await this.balancesProxy.increaseBalanceDecimals({
        id: order.id,
        userId: order.userId,
        assetId: pair.baseAssetId,
        value: new Decimal(order.volume).neg().toString(),
      });

      if (isError(result)) {
        if (result.error.code === 'NEGATIVE_BALANCE') {
          this.logger.warn('Insufficient balance for limit ask order', { ... });
          return { error: result.error };
        }
      }

      return this.conductorProxy.addOrder({ ... });
    }

// conductor.proxy.ts
    const command = createPlaceOrderCommand({
      ...payload,
      role: 'taker',
    });
    return this.rabbit.request({
      exchange: MATCHING_COMMAND_EXCHANGE,
      routingKey: matchingCommandQueue(command.partition),
      payload: command,
    });

Two details are easy to miss. First, this.calculatorProxy.addOrder(...) is not awaited, and addOrder does not return the promise from publish. With the default confirm channel, that promise resolves when the broker has confirmed the message. Nobody waits for it, so the customer gets an order id before RabbitMQ has accepted anything. Second, the calculator's reply from the conductor is ignored. It uses request anyway, because the add-order message is only acknowledged when the handler returns. Waiting for the conductor's reply means the event stays unacknowledged until the engine has committed the order, and if the engine call fails the event goes back to the queue. That is a legitimate reason to use request/reply even when you do not need the answer.

The useful question for each edge is whether the caller cannot continue without an answer that only the callee can compute at that moment. If yes, request/reply is the right tool. If the producer is reporting something that already happened, it is an event. If the caller needs data that someone else owns but that rarely changes, neither is ideal, and a local copy is usually better.

The pattern in the table: the RPCs that fit are the ones that change state in the callee and need its verdict, like reserving funds and submitting to the engine. The RPCs that fit badly are the ones that read slow-changing reference data (pairs, assets, fee tiers) owned by another service. Those reads are also the most frequent calls in the system, and the next two sections are about what they cost.

Temporal coupling means two services have to be up and responsive at the same moment for work to make progress. Every RPC creates it. A chain of RPCs creates it across every service in the chain, and the failure modes come from how the timeouts and retries combine.

Timeouts do not nest. Every hop has its own independent ten-second budget. The customer's HTTP request waits up to ten seconds on assets. The calculator's handler waits up to ten on balances and then up to ten on the conductor. Nothing passes a deadline down the chain, so a callee has no idea its caller has already given up.

Retries multiply under load. Suppose the conductor falls behind. It runs as a single active consumer with a prefetch of one, so it processes one command at a time, in order, which is what a matching engine should do. The calculator's request times out after ten seconds, the handler throws, and the default error handler requeues add-order. The calculator picks it up again, re-reserves the funds, and sends a second place-order command for the same order with a new command id. The first command is still in the conductor's queue because it had no expiration. Every timeout adds one more command to the queue of the service that is already too slow.

The transport does nothing to stop a double charge here. The protection comes from the idempotency keys: every write in the chain carries a caller-chosen id that turns a repeated write into a no-op. The ledger uses that id as the primary key of the balance row:

  public async increase(dto: IncreaseBalanceDto): Promise<BalanceSnapshot> {
    const balance =
      (await this.repository.findOne({
        where: { userId: dto.userId, assetId: dto.assetId },
        order: { version: 'DESC' },
      })) ??
      BalanceSnapshot.create({ ... });
    try {
      const increasedBalance = balance.increase(dto.id, dto.value);
      await this.repository.insert(increasedBalance);
      await this.messages.balanceUpdate({ userId: dto.userId });
      return increasedBalance;
    } catch (e) {
      if (e.message.includes('balances_balance.PRIMARY')) return balance;
      if (e.message.includes('balances_balance.balance_version_unqiue'))
        throw new BalancesException(BalancesExceptionCode.VersionError);
      throw e;
    }
  }

The reservation uses the order id, settlement uses deal.id + '-maker' and ${deal.id}-taker, and refunds use ${order.id}-leftover. A replay inserts a row whose primary key already exists, and the service returns the current balance as if it had succeeded. On the engine side, the conductor records every accepted order id in Redis and skips a second place command for the same order even when the command id is new. The version index has a side effect worth knowing about: two concurrent writes for the same user and asset collide on (userId, assetId, version), the loser throws VERSION_ERROR, the RPC handler rethrows, and the requeue default retries it. The library's error default ends up working as an optimistic-concurrency retry loop, and nobody designed it to.

A synchronous call inside the single-writer loop. The conductor processes one command at a time per partition. Cancelling an order that is not in Redis falls back to asking users:

    let order = await this.ordersStorage.getById(orderId, partition);
    if (!order) order = await this.usersProxy.getOrderById(orderId);
    if (!order || order.closedTimestamp) return;

getOrderById is rpc:users:get-order, the same handler users-ws uses, which formats the order for display by calling assets and aggregates. So on that path, matching for every pair in the partition waits on users, assets, and aggregates. The cancel itself came from users, whose HTTP handler is awaiting the conductor. It is a cycle that works while every service is healthy. When any of the three is slow, the matching engine is slow.

Settlement depends on users. When a trade executes, calculator consumes messages:deal:create and, for each side that belongs to a customer, calls rpc:users:get-user-fees and then credits the balance. If users is restarting, deal settlement stops, and every failed attempt is requeued straight away. The trade itself is committed in the engine, so nothing is lost, but customers see filled orders with balances that have not moved.

Fan-in is the number of services that depend on one service. Seven services send requests to rpc:assets:*: admin, aggregates, api, bots, calculator, users, and users-ws. Users' assets proxy alone contains fourteen request calls. The two busiest endpoints, get-one-pair-by and get-one-by, each have six call sites across the codebase. Assets holds reference data: which coins exist, which networks they move on, and each pair's tick size and limits. That data changes when an admin edits it, which is rare. It is read on every order placement, twice. It is read once per order when users renders an order list, next to a call to aggregates for that order's deals, so a page of ten orders is twenty RPCs in parallel. It is read on every WebSocket deal push.

The worst multiplier is the portfolio view. Every balance change publishes messages:balances:balance-updated with nothing but a user id. users-ws receives it and asks users to recompute the aggregated balances, and users does this:

    const usdtAsset = await this.assetsProxy.getAssetBy({
      fieldName: 'code',
      value: 'USDT',
    });

    const availables = await this.balancesProxy.getBalances(payload);
    ...
    for (const { assetId, decimalValue } of availables) {
      ...
      const equivalent = await this.spotService.convert({
        fromAssetId: assetId,
        toAssetId: usdtAsset.id,
        amount: dValue.toString(),
      });
      ...
    }
    ...
    for (const order of orders) {
      const pair = await this.assetsProxy.getPairBy({
        fieldName: 'id',
        value: order.exchangePairId,
      });
      ...
        const equivalent = await this.spotService.convert({ ... });

spotService.convert is rpc:calculator:convert. So one balance change costs one RPC to assets, one to balances, one conversion per asset held, one pair lookup plus one conversion per open order, and more for staking, launchpad, and futures holdings. Every call is awaited inside a for loop, so they run one after another.

Exactly one service caches any of this. The aggregates assets proxy keeps pairs in the NestJS cache manager for ten minutes:

    const chached = await this.cacheManager.get<Pair>(
      `pair:${payload.fieldName}:${payload.value}`,
    );
    if (chached) {
      return chached;
    }
    ...
    await this.cacheManager.set(
      `pair:${payload.fieldName}:${payload.value}`,
      newCache.data,
      600000, // 10 min
    );

Nothing invalidates it. When an admin updates a pair, apps/admin/src/pairs/pairs.service.ts publishes messages:calculator:resetConvertationGraphs and nothing about the pair itself, so aggregates can serve stale limits for up to ten minutes. Even so, this cache is the only thing standing between aggregates and one assets RPC per deal it records.

These are the problems I can point to in the code, in the order I would fix them.

Order submission is fire-and-forget. Users drops the publish promise and passes no persistent flag. If the publish fails or is never confirmed, nothing in users finds out, and the customer holds an order id for an order that does not exist. If the broker restarts before calculator consumes the message, a non-persistent message in a classic queue is gone. The fix is small: await the publish, set persistent: true, and return an error to the HTTP caller if the broker does not confirm.

Rejected orders disappear. When the reservation returns NEGATIVE_BALANCE, calculator logs a warning and returns. No event goes out, so users never stores the order and users-ws never pushes anything. The customer saw a success response with an id and then nothing. The shared matching contracts already define order.rejected.v1. Calculator should publish it with the order id and the reason, and users should record the order as rejected.

Failures requeue forever. A request for a pair that was deleted makes pair.baseAssetId throw inside calculator, the message is requeued, and it fails again at full speed until someone purges the queue. KYC tried to opt out:

  @RabbitRPC({
    exchange: KycExchanges.getKyc,
    routingKey: KycExchanges.getKyc,
    queue: KycExchanges.getKyc,
    errorBehavior: MessageHandlerErrorBehavior.ACK,
  })
  async getKyc({ userId }: GetKycDto) {
    return this.kycService.findKycByUserId(userId);
  }

Look back at the error handler selection in setupRpcChannel. this.config.defaultRpcErrorHandler is checked before rpcOptions.errorBehavior, and the library's defaults always define it, so in 5.7.0 errorBehavior on an @RabbitRPC handler has no effect. The KYC handlers still requeue. What I would do: set defaultRpcErrorHandler to reject without requeue, because the caller will time out anyway and retrying is the caller's decision. For subscribers, move the queues to quorum queues with a delivery limit and a dead-letter exchange, so a message that fails five times lands somewhere a human can see it.

Timeouts are not a budget. Every request should set expiration equal to its timeout, so an abandoned request dies in the queue instead of running late. The HTTP path should have the shortest deadline and pass what is left down the chain in a header, and each callee should drop work whose deadline has passed.

Thin events turn into callbacks. balance-updated with only a user id forces every consumer to call back, which makes the event an RPC with extra steps. The event should carry the new balance for that asset. The futures service already does this for market data: apps/futures/src/services/futures.messages.ts publishes full klines, pairs, and trades with { expiration: 60 * 1000 }, which is the right shape for data that is useless a minute later.

Reference data should be local. Assets should publish pair-updated and asset-updated events with the full record. Each consumer keeps a copy in memory, loads it with one snapshot RPC on startup, and applies events after that. The same goes for fee tiers in calculator. This removes the most frequent RPCs in the system and takes assets off the order path entirely.

No RPC inside the engine loop. If an order is not in the partition's Redis state, a cancel should be rejected, not resolved by asking users. The engine would then have no dependency on any service outside its own partition while it runs a command.

Generate the map. Every edge is already declared in code, in @RabbitRPC and @RabbitSubscribe decorators and in the proxy classes. A script that reads those and writes the Mermaid file in CI would have caught all three errors in the hand-drawn map.

Some limits come from the conventions themselves, so fixing individual calls does not remove them.

• users-ws is one process. It has no Socket.IO adapter, so each process only knows its own sockets, and some of its event queues have no suffix. Run a second replica and those queues split messages between the two, so a customer connected to replica A misses the half that went to B. Adding suffixes gives every replica every message, which works, but then every replica makes the callback RPCs for every event. Scaling it needs a queue per instance and a way to deliver only to the instance that holds the user's socket.
• Direct reply-to ties replies to one channel. If a caller restarts while a request is in flight, the reply has nowhere to go and the callee's side effect has already happened. That is survivable only because the writes are idempotent and the callers retry. Any new write endpoint that forgets an idempotency key will double-apply on the first restart.
• Reads fail with the broker. Because reference data is fetched by RPC, a RabbitMQ outage stops reads as well as writes. Services with a local read model would keep validating orders and rendering pages from memory while the broker recovers.
• Throughput is prefetch divided by latency. The default prefetch is ten messages per channel, and handlers await their RPCs one after another. A settlement handler making four or five sequential RPCs per trade can only settle as fast as those round trips allow, and raising prefetch just moves the queue into the callees.
• The broker is one failure domain. Every edge in the system, commands, events, and reads, goes through the same cluster. Its capacity and availability are the ceiling for the whole backend.

This is the rule I would hand to the team now, in the order I would ask the questions for each new edge:

1. Is it data another service owns that changes much less often than you read it? Pairs, assets, fee tiers, feature flags. Keep a local copy fed by events, with one snapshot request on startup. Do not call for it per request.
2. Are you reporting something that already happened? Publish an event. Put the data consumers need in the payload, include an id they can deduplicate on, and set a time to live if the data goes stale quickly.
3. Do you need a verdict that only the owner can compute right now, before you can continue? Reserving funds, submitting to the engine. Use request/reply, with an idempotency key, expiration equal to the timeout, no requeue on handler errors, and a deadline shorter than your own caller's.
4. Is the caller an HTTP handler or a single-writer loop? Try hard to take the RPC out. Every RPC there adds the callee's worst latency to your user's wait or to the engine's cycle.
5. Are you about to publish an event with only an id so consumers can call you back? Either put the data in the event or give consumers a read model. An id-only event followed by a callback keeps every downside of the RPC and adds the queue delay on top.

Applied to Bitsten, rules one and five remove most of the RPC traffic: pair and asset lookups, fee lookups, and the portfolio callbacks. Rule three keeps the two request/reply edges that carry real decisions, the balance reservation and the matching command, and makes them safer. The customer's order would then touch assets zero times, and settlement would keep running while users is being deployed. The comparison below is the same rule in table form.