Blast-Radius Containment in a Multi-Tenant Engine

One tenant points a workflow at an endpoint that started returning 500s. Their steps fail and retry. Retries consume claim slots, connections, and worker capacity. Every other tenant's workflows slow down, then queue, then miss their deadlines. Nobody did anything wrong except the one customer whose vendor had a bad afternoon.

This is not exotic. It is what a shared engine does by default, and preventing it means deciding — for every shared resource — what the tenant boundary is and enforcing it there. Four mechanisms cover most of the surface, and each has a detail that is easy to get subtly wrong.

A circuit breaker stops calling a dependency that is clearly broken: count failures, trip open past a threshold, reject fast during a cooldown, then probe. Standard. The part that decides whether it helps or hurts in a multi-tenant engine is the key.

Key it by handler name alone and you have built a weapon. Tenant A's `http_request` steps fail against their broken endpoint, the breaker trips, and now every tenant using `http_request` — the most common handler in the system — is rejected. One customer's outage becomes a platform outage, delivered by the reliability feature.

/// Per-tenant isolation. The registry is keyed by `(TenantId, String)` so a
/// failing handler for one tenant cannot trip the breaker for any other
/// tenant that shares the same handler name.
pub struct CircuitBreakerRegistry {
    breakers: DashMap<Key, CircuitBreakerState>,
    default_threshold: u32,
    default_cooldown_secs: u64,
}

That key is a `(TenantId, String)` pair, and looking one up naively means allocating a `Key` on every check — on the hot path, for every step. The registry avoids it with a `Borrow<dyn CircuitKey>` implementation, so a lookup can be performed from borrowed `&str` components without constructing the owned key. Hash and equality are defined on the trait object so the borrowed and owned forms agree, which is the requirement `Borrow` actually imposes and the part people get wrong.

It is a small trick with a general lesson: composite-key maps in Rust invite an allocation per lookup, and `Borrow` is the standard escape hatch — the same mechanism that lets you look up a `HashMap<String, _>` with a `&str`, extended to a tuple.

An in-memory breaker resets when the process restarts. Every tripped breaker returns to Closed, and a fleet restarting during a dependency outage collectively resumes hammering the thing that is down — at exactly the moment it is least able to cope.

So `Open` transitions are mirrored to storage and rehydrated at boot, cooldown clocks preserved. But only `Open`:

• `Closed` is the default. Persisting a row for every untripped breaker means writing rows for the overwhelmingly common case to record the absence of a problem.
• `HalfOpen` is a transient probe state owned by the live process. Persisting it across a restart would restore a probe that no process is running.
• `Open` is the only state whose loss changes behaviour badly, so it is the only one that earns a write.

Persistence runs off the hot path — transitions spawn a fire-and-forget write, so `check`, `record_failure`, `record_success`, and `reset` stay synchronous and the in-memory registry's semantics are unchanged for callers. The trade is explicit and worth naming: a crash in the window between an `Open` transition and its storage write loses that breaker's state. It is a durability backstop, not a source of truth, and treating it as the latter would mean putting a database write in front of every step dispatch.

A breaker on a handler with no external dependency produces only false positives. A `fail` step in a test suite, or a `send_signal` that loses a race, should not take down every instance using that handler for a minute.

pub fn is_breaker_tracked(handler: &str) -> bool {
    !matches!(
        handler,
        "noop" | "log" | "sleep" | "fail" | "self_modify"
            | "emit_event" | "send_signal" | "query_instance" | "human_review"
    )
}

The list lives next to the breaker rather than as a flag on each handler, for a reason worth generalising: policy that must be reviewed as a set should be visible as a set. Scattered opt-outs mean nobody can answer "which handlers skip the breaker?" without grepping, and a wrong entry hides indefinitely.

When a breaker is open, the engine does not simply fail the step. Pre-flight prefers a configured `fallback_handler` if that handler's own breaker is closed, and only defers the instance until the cooldown expires when there is no usable fallback. The pre-flight is pure decision logic with no storage writes — it returns either a step definition to dispatch or a time to defer until, and the caller applies the transition appropriate to its execution path. Two dispatch paths, one decision function, so open-breaker behaviour cannot drift between them.

Breakers handle failing dependencies. They do nothing about a tenant who is simply enormous. If the scheduler claims work in pure priority-then-time order, a tenant with 500,000 queued instances fills every batch and everyone else waits — no failures anywhere, just a queue that never reaches them.

The claim query caps rows per tenant per batch using a window function over the locked candidates, and over-selects so that the cap does not starve the batch when one tenant dominates the lock pool. That interaction — a fairness rule accidentally shrinking throughput — is covered in detail in the scheduler write-up.

The subtler category is coordination between workflows. An engine that lets workflows emit events, send signals, and query each other has created a way for one tenant to reach another, and the enforcement point has to be inside the operation rather than in a wrapper:

• `emit_event` spawns a child instance in the same tenant only, with deduplication scoped to either the parent or the tenant.
• `send_signal` targets same-tenant instances only, and the terminal-state check plus the enqueue happen atomically in one storage transaction — otherwise a signal can be enqueued for an instance that completed between the check and the write.
• `query_instance` reads same-tenant instances only, returning `{ found: false }` rather than a distinguishable error for a missing target — because a different error for "exists but not yours" is a cross-tenant existence oracle.

That last one is the kind of leak that survives a security review. The access check is correct, the data is not returned, and the response still tells an attacker whether an ID exists in another tenant. Uniform responses for "absent" and "forbidden" are not paranoia; they are the difference between a check and a guarantee.

Logical isolation has a floor. Some tenants need their data in a specific jurisdiction, or on hardware their contract names, or simply away from everyone else's. That means routing a tenant to an independently configured storage backend — and routing is where a convenient default becomes a data breach.

The router is authoritative and fails closed. Every tenant must have a durable placement record naming a backend registered in the current process. A missing record returns `NotFound`. An unrecognised backend identifier returns `Unsupported`. Neither falls back to another partition, because "we could not find your placement, so we used the default one" is how a regulated tenant's data ends up in the wrong country.

Placement changes are epoch-fenced with the same reasoning as everywhere else in the system:

tenant_storage_placements
  tenant_id   TEXT PRIMARY KEY   -- the routing key
  backend_id  TEXT               -- operator-defined name, never a connection string
  epoch       BIGINT             -- positive, monotonically increasing
  updated_at  TIMESTAMPTZ

-- advance_tenant_placement: insert a first placement, or replace only when
-- the proposed epoch is strictly greater. Stale writers get a conflict.

`backend_id` being an operator-defined identifier rather than a connection string is deliberate. Placement metadata belongs in a highly available control-plane store and may be replicated more widely than credentials should be; keeping connection details in each process's protected configuration means the routing table is not a secret even though it is authoritative.

And the router is honest about its scope: it selects a partition. It does not copy data, coordinate dual writes, or make a multi-backend transaction atomic. Moving a tenant means quiescing writes, copying, validating, registering the destination everywhere, and only then advancing the epoch. A component that quietly did more than that would be a component nobody could reason about during a migration.

The generalisable version of all of this is a single question asked repeatedly: for every shared resource, what is the tenant boundary, and what happens when one tenant exhausts it?

1. Enumerate the shared resources: connection pools, worker capacity, rate limits, caches, breakers, queues, scheduler batches, memory.
2. For each, decide whether the boundary is per-tenant, global, or tiered — and write the decision down, because an unstated boundary defaults to global.
3. Make cross-tenant operations impossible inside the operation, not in a wrapper that a new code path can bypass.
4. Return identical responses for "does not exist" and "not yours". Different errors are an existence oracle.
5. Fail closed on routing. A default backend is a convenience that becomes a compliance incident exactly once.
6. Test the noisy neighbour explicitly: run one tenant at ten times the load of the rest and assert the others still meet their objectives. Isolation you have not tried to break is a hypothesis.

The reason to do this early is that isolation is nearly impossible to retrofit. Every one of these boundaries has to exist at the point the resource is consumed, and by the time a shared engine has fifty call sites, adding a tenant dimension to each of them is a migration rather than a patch. Deciding the boundary while there are five call sites costs almost nothing.