Exchange Balance Ledger Design: How Bitsten Keeps Balances From Going Negative

Every exchange answers the same question thousands of times a second: can this user spend this amount right now, and if so, how do we make sure the same money cannot be spent twice. This is how the Bitsten backend answers it, using the real code, including the parts I am not proud of.

Some names first, since the rest of the article uses them. Bitsten is a NestJS monorepo of small services that talk over RabbitMQ, a message broker. When I say one service makes an RPC to another, I mean it publishes a request message and waits for a reply message, with a 10 second timeout by default. The services that matter here:

• users: the HTTP API that clients call. It creates orders and withdrawals and shows balances.
• calculator: the accountant. It turns business events (order placed, trade executed, order closed, deposit confirmed) into signed amounts to add to a balance.
• balances: owns one MySQL table and does exactly one kind of write: add a signed amount to one user's balance of one asset, refusing if the result would be negative.
• conductor: the matching engine service. It keeps the order books, matches incoming orders against resting ones, and emits trade and order-closed events.
• custody: talks to the wallet provider that holds the actual coins on chain.

A balance ledger for an exchange has to hold five properties, and it helps to name them before looking at code:

1. No balance ever goes below zero.
2. No lost update: if two changes to the same balance happen at the same moment, the result reflects both of them, never only the one that wrote last.
3. Each business event is applied at most once, even when the broker delivers its message twice.
4. Money promised to an open order cannot be withdrawn or promised to a second order.
5. When an order ends, every unit that was set aside for it has either been paid to a counterparty or returned to the owner.

Four of these hold in the code as written. The fifth does not in one specific case, and I will show the line.

The whole balances service is built on one table. There is no UPDATE anywhere in it. Every change inserts a new row carrying the resulting balance and a version number one higher than the previous row for the same user and asset.

CREATE TABLE `balances_balance` (
  `id` varchar(255) NOT NULL,
  `userId` int NOT NULL,
  `assetId` int NOT NULL,
  `value` varchar(255) NOT NULL,
  `version` int NOT NULL,
  `createdAt` datetime NOT NULL,
  UNIQUE INDEX `balance_version_unqiue` (`userId`, `assetId`, `version`),
  PRIMARY KEY (`id`)
) ENGINE=InnoDB

value is a string holding an integer count of 10^-18 units of the asset. The controller converts decimal amounts with new BigNumber(payload.value).shiftedBy(18).integerValue(BigNumber.ROUND_FLOOR) and the entity does arithmetic in JavaScript BigInt. That avoids floating point entirely, which is the first thing any ledger has to get right. Floor rounding means a credit with more than 18 decimals rounds down and a debit rounds to a slightly larger magnitude, so any dust lands on the exchange side.

The arithmetic itself lives on the entity:

increase(id: string, value: bigint): BalanceSnapshot {
  const currentValue = BigInt(this.value);
  const newValue = value + currentValue;

  if (newValue < 0) {
    throw new BalancesException(BalancesExceptionCode.NegativeBalance);
  }

  const balance = Object.assign(new BalanceSnapshot(), {
    id,
    assetId: this.assetId,
    userId: this.userId,
    value: newValue.toString(),
    createdAt: new Date(),
    version: this.version + 1,
  });
  return balance;
}

It is worth being clear about what this table is. People call it "the ledger", and the earlier requirements article on this site does too. Structurally it is a versioned history of balances, one account per (user, asset) pair. Each row stores the balance after a change. You can recover the delta by diffing two consecutive versions. You cannot recover why it changed except from the naming convention of the id column, which you will see below is things like an order id, <deal id>-maker, or refund-<withdrawal id>. And each change touches exactly one account, which makes it single-entry bookkeeping. That has consequences I come back to in the fee and deposit sections.

Here is the service method that every balance change in the system goes through. Read it with two concurrent requests in mind.

public async increase(dto: IncreaseBalanceDto): Promise<BalanceSnapshot> {
  const balance =
    (await this.repository.findOne({
      where: { userId: dto.userId, assetId: dto.assetId },
      order: { version: 'DESC' },
    })) ??
    BalanceSnapshot.create({
      id: dto.id,
      userId: dto.userId,
      assetId: dto.assetId,
    });
  try {
    const increasedBalance = balance.increase(dto.id, dto.value);
    await this.repository.insert(increasedBalance);
    await this.messages.balanceUpdate({ userId: dto.userId });
    return increasedBalance;
  } catch (e) {
    if (e.message.includes('balances_balance.PRIMARY')) return balance;
    if (e.message.includes('balances_balance.balance_version_unqiue'))
      throw new BalancesException(BalancesExceptionCode.VersionError);
    throw e;
  }
}

There is no lock and no transaction. The method reads the latest row, computes the new value in memory, and inserts. The classic race is two withdrawals of 30,000 USDT against a 40,000 USDT balance arriving together. Both read version 7 with 40,000. Both compute 10,000, which is not negative, and both try to insert version 8. With a plain UPDATE balance SET value = ? the second write would silently overwrite the first and the user would have spent 60,000 out of 40,000.

The unique index on (userId, assetId, version) is what stops it. Only one insert of version 8 can succeed. The other fails with a duplicate key error on balance_version_unqiue and the service turns that into VersionError. This is optimistic concurrency control: nobody holds a lock while computing, and the conflict is detected at the moment of writing. The index acts as a compare-and-set: "write version 8 only if nobody else has". Because the negative check always runs against the exact row being replaced, a debit can never pass the check against a stale balance and still land.

What happens to the loser? The controller only turns NegativeBalance into an error reply; everything else is rethrown. The retry comes from the RabbitMQ library defaults, which I did not appreciate until I read them:

const defaultConfig = {
    name: 'default',
    prefetchCount: 10,
    defaultExchangeType: 'topic',
    defaultRpcErrorHandler: (0, errorBehaviors_1.getHandlerForLegacyBehavior)(errorBehaviors_1.MessageHandlerErrorBehavior.REQUEUE),
    defaultSubscribeErrorBehavior: errorBehaviors_1.MessageHandlerErrorBehavior.REQUEUE,
    ...
    defaultRpcTimeout: 10000,

A thrown error in an RPC handler requeues the request: RabbitMQ puts the message back and delivers it again, and on the next attempt findOne sees version 8 and the retry computes against the new value. prefetchCount: 10 is also why the race happens in practice. Each balances instance processes up to ten requests at once, so two changes for the same user can interleave.

The second catch branch handles duplicates. The row id is the primary key, and every caller sets it to the id of the business event: the order id when funds are reserved, <deal id>-taker when a trade is settled, and so on. If the same event is delivered twice, the second insert collides on PRIMARY and the service returns success without changing anything. That is the idempotency key: a caller-chosen id that makes repeating a request harmless.

I like one property of this more than the rest. The matching engine docs require every durable consumer to keep an inbox, a table of processed event ids written in the same database transaction as the business change. The runbook puts it this way: "Use AMQP messageId / x-event-id as the consumer inbox key and commit it in the same database transaction as the projection." The balances table gets that for free because the idempotency key and the balance change are the same row. There is no window in which one is committed and the other is not. And because the key is derived from business ids (a deal id is <taker order id>:<maker order id>, deterministic), even a republished event with a fresh event id is deduplicated.

Now the concrete request. A user holds 40,000 USDT and places a limit buy of 0.5 BTC at 60,000 on the BTC/USDT pair. In exchange terms BTC is the base asset (the thing being bought), USDT is the quote asset (the thing you pay with), and a buy is a bid. A limit order says "buy up to 0.5 BTC, paying at most 60,000 per BTC".

Step 1, users service. The HTTP handler generates the order id, validates the pair and amount, publishes a message for the calculator, and returns the id to the client immediately.

const id = uuid();
...
await this.validateLimitOrderCreateInfo(exchangePair, dto);

this.calculatorProxy.addOrder({
  id,
  userId,
  type: OrderType.limit,
  volume: dto.amount,
  side: dto.side,
  pairId: exchangePair.id,
  timestamp: Date.now(),
  price: dto.rate,
});
...
return id;

Note that addOrder is a fire-and-forget publish, so the client gets an id before anyone has checked the balance. Hold that thought.

Step 2, calculator reserves the funds. For a limit bid the calculator debits volume * price of the quote asset, using the order id as the idempotency key. Only if that succeeds does it forward the order to the matching engine.

if (order.type === 'limit' && order.side === 'bid') {
  const result = await this.balancesProxy.increaseBalanceDecimals({
    id: order.id,
    userId: order.userId,
    assetId: pair.quoteAssetId,
    value: new Decimal(order.volume).mul(order.price).neg().toString(),
  });

  if (isError(result)) {
    if (result.error.code === 'NEGATIVE_BALANCE') {
      return;
    }
  }

  return this.conductorProxy.addOrder({
    id: order.id,
    ...
  });
}

Step 3, balances. It reads version N (40,000 USDT), computes 40,000 minus 30,000, inserts version N+1 with 10,000 USDT and id equal to the order id, and publishes a balance-updated message so the user's WebSocket refreshes.

Step 4, conductor. The calculator sends a place-order command to the matching engine partition that owns BTC/USDT. From here the order is in the book.

The design decision hiding in step 2 is that there is no "locked" or "held" balance anywhere. Reserving funds is an ordinary debit. The 30,000 USDT leaves the balances table and, as far as the ledger knows, exists only as an open order inside the matching engine. Property 4 from the list holds trivially: money that is not in the balance cannot be withdrawn or reserved again, and the version index already guarantees two reservations cannot both pass the negative check.

There is a real benefit to this, which shows up in the next section. There is also a small bug in the block above. When a bid fails for insufficient funds, the calculator returns undefined without logging or notifying anyone. The ask branch of the same method logs a warning and returns the error. Either way, the user already got an order id back in step 1 for an order that will never exist.

Say the book has a resting sell (an ask) for 0.2 BTC at 59,900. The resting order is the maker, because it was there first and made liquidity. Our incoming buy is the taker. The matching engine computes the trade, which the code calls a deal, at the maker's price:

const takerQuoteVolumeChange = takerVolumeChange
  .mul(maker.price)
  .toFixed(maker.quoteDecimals, quoteRounding(taker));
...
return {
  matched: true,
  deal: {
    id: `${taker.id}:${maker.id}`,
    time: now,
    maker,
    taker,
    makerVolumeChange: makerVolumeChange.toString(),
    takerVolumeChange: takerVolumeChange.toString(),
    makerQuoteVolumeChange,
    takerQuoteVolumeChange,
  },
};

So the deal is 0.2 BTC for 11,980 USDT. That is a partial fill: 0.3 BTC of our order is left, and it now rests in the book as a maker at 60,000. The engine publishes the deal event, and the calculator settles it:

const isTakerUser = !!deal.taker.userId;
// if maker is user increase balance
if (isTakerUser) {
  const assetId =
    deal.taker.side === 'ask' ? pair.quoteAssetId : pair.baseAssetId;

  const value =
    deal.taker.side === 'ask'
      ? deal.takerQuoteVolumeChange
      : deal.takerVolumeChange;

  const userFees = await this.usersProxy.getUserFees({
    userId: deal.taker.userId,
  });

  takerFee = new Decimal(value).mul(userFees.taker).div(100);

  await this.balancesProxy.increaseBalanceDecimals({
    id: `${deal.id}-taker`,
    userId: deal.taker.userId,
    assetId,
    value: new Decimal(value).sub(takerFee).toString(),
  });
}

Settlement only touches the asset each side receives. Our buyer receives 0.2 BTC minus the taker fee. The seller (if it is a user) receives 11,980 USDT minus the maker fee, with id <deal id>-maker. The side each party pays is not touched, because it was already debited when that order was placed. The fee is a percentage from the users service, looked up at settlement time, and it is taken out of what you receive.

This is where reserve-by-debit pays off. After placement, every ledger operation for an order is a credit: fills credit the receiving asset, and closing the order credits back whatever is left. Credits cannot make a balance negative, and adding numbers is order-independent. Deal events and order-closed events travel on different RabbitMQ queues and can be processed in any order, concurrently, or twice, and the final balance is the same. The only operations that can fail on funds are the debits at the edges: placing an order, withdrawing, transferring out. That makes ordering between queues a non-issue for correctness, which in a system where one trade fans out to several consumers is worth a lot.

Settlement is also retry-safe in the way you want. Both credit calls are awaited. If the taker credit times out, the handler throws, RabbitMQ requeues the deal event, the maker credit on the second pass collides on its primary key and is skipped, and the taker credit is tried again.

Now the part that is plainly wrong. Where does the fee go? Nowhere in the ledger. The buyer is credited 0.2 BTC minus fee, the seller's BTC was debited at placement, and the fee amount is not credited to any exchange account. It is only published to the aggregates service for reporting, with a publish that is not awaited. The same is true for the other side of trades against the platform's own market-making bots, which carry no userId: isMakerUser is false, no ledger row is written for the bot, and the user's credit has no matching debit anywhere. The sum of all balances in this table is therefore not something you can reconcile against the coins custody actually holds. More on that in the double-entry section.

The remaining 0.3 BTC sits in the book. The user cancels it. The cancel goes straight from the users service to the matching engine, which removes the order and emits an order-closed event carrying the order's final state: volume (what is left unfilled), price, and quoteVolume (the quote amount actually paid so far). The calculator refunds the unused reservation:

const id = `${order.id}-leftover`;
...
if (order.side === 'bid') {
  // For bid orders, refund the quote asset amount
  if (order.type === 'limit') {
    ...
    // For limit orders, refund remaining volume * price
    const quoteAmount = new Decimal(order.volume).mul(order.price);
    return this.balancesProxy.increaseBalanceDecimals({
      id,
      userId: order.userId,
      assetId: pair.quoteAssetId,
      value: quoteAmount.toString(),
    });
  }

  if (order.type === 'market') {
    ...
    // For partially filled market orders, refund unused amount
    const unusedAmount = new Decimal(order.quoteLimitVolume).sub(
      order.quoteVolume,
    );

The refund id <order id>-leftover makes a duplicate close event harmless. The refund for our order is 0.3 times 60,000, which is 18,000 USDT. Now add it up:

Twenty USDT is gone. It is the price improvement: the buyer was willing to pay 60,000 and got 59,900 on 0.2 BTC, a 20 USDT saving that was reserved but never spent and never returned. If the order had filled completely, the refund would be 0 times 60,000 and the whole improvement would vanish. This only happens to limit bids that execute as takers against asks priced below their limit, because makers always trade at their own price and ask orders reserve base volume, which is consumed exactly. It violates property 5 from the list at the top.

The market bid branch right below it shows the correct formula: refund what was reserved minus what was actually paid. For a limit bid that is volumeTotal * price - quoteVolume, and the close event already carries all three fields. It is a one-line fix. The reason the bug is easy to write is the design: once the reservation is a plain debit, the ledger has no record of how much was reserved for this order, so the refund has to be reconstructed from order fields, and the reconstruction picked the wrong ones.

Users expect to see three numbers per asset: available, in orders, and total. Since the balances table only holds what is spendable, the users service rebuilds the rest on every request. The variable name gives the model away:

const availables = await this.balancesProxy.getBalances(payload);
...
const orders = await this.ordersService.getOpenOrdersForUser(
  payload.userId,
);

for (const order of orders) {
  ...
  if (volumeFilled.gt(0)) {
    price = new Decimal(order.quoteVolume)
      .div(volumeFilled)
      .toFixed(priceDecimals);
  } else {
    price = order.price;
  }
  if (price)
    quoteVolumeTotal = new Decimal(order.volumeTotal)
      .mul(price)
      .toFixed(quoteDecimals);
  ...
  const dValue = new Decimal(quoteVolumeTotal).minus(order.quoteVolume);

For a bid that has partly filled, "in orders" is the remaining volume times the average fill price. In our example that is 0.3 times 59,900, or 17,970, while the calculator will actually refund 18,000 when the order closes. So there are now three different answers to "how much USDT is tied up in this order": 18,000 (what refund will return), 17,970 (what the screen shows), and 18,020 (what was really withheld and not spent). A ledger with an explicit held balance has one answer, stored, and the screen reads it.

Withdrawals follow the same pattern as orders: debit first, act second. The calculator debits amount + fee with the withdrawal id as the key. If the balance would go negative, it returns. Otherwise it asks custody to send the coins (immediately in auto mode; in manual mode, later, when the transaction is moved from CREATED to PENDING) and records a transactions row with status CREATED. If custody later reports the withdrawal as failed, the calculator credits amount + fee back with the id refund-<withdrawal id>, so a repeated failure report cannot refund twice. The fee part of the debit, as with trading fees, is not credited to any exchange account.

Deposits run the other way. Custody sends status updates, and when a deposit is marked complete the calculator credits it, keyed by the custody transaction id:

this.upsertOne(transaction);

if (updates.isCompleted) {
  this.balancesProxy.increaseBalanceDecimals({
    id: updates.id,
    userId: updates.userId,
    assetId: asset.id,
    value: amount.toString(),
  });
}

Neither call is awaited. The handler returns, the RabbitMQ message is acknowledged, and the balance credit is still in flight. If the balances service is down or the RPC times out, the rejected promise goes nowhere, the message is already gone, and the user's deposit is recorded as complete in transactions but never reaches their balance. The only thing that would repair it is custody sending another update for the same deposit, which then credits correctly because the id deduplicates. The failed-withdrawal refund in handleCustodyWithdrawUpdates has the same missing await. Compare that to trade settlement, which awaits both credits and therefore gets retried by the broker when a credit fails. Both paths use the same ledger and the same idempotency keys; the deposit path simply never lets the error reach RabbitMQ.

Spot-to-futures transfers in the users service are a small saga (a multi-step operation where a failed step is undone by a compensating step): debit spot, fund futures, and on failure credit spot back with <id>-refund. If the process dies between debit and compensation, nothing retries it, because the id is a fresh uuid() held only in memory.

The core write path is sound: integer amounts, append-only rows, a unique index as compare-and-set, and business ids as idempotency keys. The flaws are about what the ledger does not record. In rough order of how much money they protect:

1. Fix the limit bid refund today. Refund volumeTotal * price - quoteVolume, the same shape as the market bid branch. Then write a one-off query that finds closed limit bids where the refund plus the quote actually paid is less than the reservation, and credit the difference with an id like <order id>-improvement so the correction is itself idempotent.
2. Make every change a balanced journal entry. In double-entry bookkeeping each business event writes two or more postings to different accounts, and the postings for each asset sum to zero. A fee becomes a credit to an exchange fee account. A trade against a bot debits a liquidity account. A deposit debits a custody clearing account that mirrors coins on chain. The invariant "sum of all postings per asset is zero" becomes a query you can run, and "user liabilities equal custody holdings" becomes checkable.
3. Give each user an explicit held account per asset. Placing an order moves funds from available to held. A fill consumes from held. Closing an order releases whatever held amount is left for that order, read straight from the postings. The price improvement bug cannot be written against that model, and the "in orders" number on the screen becomes a stored value.
4. Stop detecting duplicates by searching error text. The service decides between "already applied" and "concurrent write" with e.message.includes(...) on a table name and a misspelled index name. Renaming either turns every duplicate into an unrecognized error, which the RPC handler requeues forever. I would check MySQL error 1062 plus the key name from the structured error, or insert into a separate processed-events table inside the same transaction and branch on affected rows.
5. Await every posting. Deposits and withdrawal refunds should fail loudly and requeue like trade settlement does. A deposit handler that returns before the credit is confirmed is the single most likely way for this system to lose customer money.
6. Store the reason next to the amount. Add reason (order-reserve, fill, fee, refund, deposit, withdrawal) and referenceId columns so audit queries stop parsing ids.
7. Reject at the edge. Check the available balance synchronously in the users service before returning an order id, keep the calculator debit as the authoritative check, and send a rejection event when it fails so the client learns why its order never appeared.

For the same 0.5 BTC example, the postings in a double-entry version would look like this. This is a proposal; the codebase has nothing like it yet:

Even with those fixes, the one-table, optimistic-concurrency approach has limits I would plan for.

• Hot accounts. Every write to a (user, asset) pair competes for the next version number. A user whose orders fill many times a second, or a shared exchange fee account in the double-entry version, turns into a queue of version conflicts, and each conflict costs a full RabbitMQ requeue and redelivery which is far slower than an in-process retry. The usual fix for a shared account is to split it into N sub-accounts (for example one per balances instance) and sum them when reporting.
• Reads over history. The current balance is "the row with the highest version". The per-user query does a MAX(version) group-by over that user's whole history, and getGroupedLast, which sums every user's latest balance per asset, scans every balance row ever written. Both get slower forever. A separate current-balance table updated in the same transaction as the history insert, or periodic checkpoints, would bound that.
• Cross-service atomicity. Orders, transfers, and withdrawals already span two or three services, and the only glue is idempotent retries and compensations. That works while every step is retried until it succeeds. It stops working the moment a step is fire-and-forget, as the deposit path shows. Double-entry inside one database transaction also needs both accounts in the same database, which fights with sharding by user.
• Reservation outside the engine. Funds are reserved by the calculator before the matching engine sees the order, so the ledger relies on a close event coming back for every order the engine drops. At higher volume, a common next step is moving the available-balance check into the matching engine itself, which already processes each market sequentially, and making the ledger the downstream record of what the engine decided.
• Reconciliation. Without balanced entries there is no internal way to prove that customer balances equal the coins in custody. That check has to happen outside the ledger, against custody reports, which is slow and manual. This is the strongest single reason to move to double entry before scaling anything else.

For a small or mid-sized spot exchange, the parts of this design worth copying are the integer amounts, the append-only history, the unique version index, and deriving idempotency keys from business ids instead of message ids. The parts I would not copy are single-entry postings, reservations modeled as plain debits, and any handler that acknowledges a message before its balance change is confirmed.