Moving a Running Workflow Across a Trust Boundary

Three situations make this worth the complexity, and none of them are solved by restarting the workflow somewhere else.

• **Connectivity.** A workflow drives an onboarding flow on a phone. The device goes offline mid-flow. The execution should continue on-device and reconcile when the network returns — not stall waiting for a server it cannot reach.
• **Data residency.** A step must process data that cannot leave a jurisdiction or a customer's own infrastructure. Only that one step needs to run there; the surrounding orchestration does not.
• **Organisational boundary.** A workflow spans two companies. Neither will run the other's code or expose its database, but the process is genuinely one process and needs one continuous record.

The naive approach — serialise state, send it, deserialise, continue — works right up until the source does not actually stop. Then two runtimes believe they own the same execution, and both call the payment API. Everything that follows exists to make that impossible.

A boolean "is_owner" cannot survive a partition: the old owner still believes its own flag. What is needed is a monotonically increasing generation number, so that a stale owner's writes are recognisably stale rather than merely late.

pub struct ExecutionEpoch(u64);

pub struct ContinuityExecution {
    pub continuity_id: ContinuityId,     // stable across every move
    pub tenant_id: TenantId,
    pub current_instance_id: InstanceId, // the local run, changes per runtime
    pub owner_runtime_id: RuntimeId,     // who may act right now
    pub epoch: ExecutionEpoch,           // fencing token
    pub state: OwnershipState,           // Owned | Transferring | Completed
}

Every handoff increments the epoch. An operation arriving with an epoch lower than the stored one is rejected — not retried, not queued, rejected. This is the fencing-token pattern Martin Kleppmann described for distributed locks, applied to execution authority rather than to a lock: the resource itself checks the token, so a delayed request from a previous owner cannot take effect no matter how convinced that owner is.

Two details matter more than they look. The epoch increment is checked, returning an error on overflow rather than wrapping — a wrapped epoch would silently re-authorise every stale owner in history. And `Transferring` is a distinct state from `Owned`, so there is an interval where neither side may dispatch effects. A handoff that crashes halfway leaves the execution provably parked rather than ambiguously owned by both.

A database constraint backs this at the storage layer: a unique index on `(tenant_id, current_instance_id)` means a local instance can belong to exactly one continuity execution. Application logic enforcing this is a check that can be forgotten; a unique index is a guarantee.

This is where naive migration fails hardest. Say a workflow charges a card on the server, then relocates to a device. If the effect record does not travel — or travels but is not authoritative — the device may re-run the step and charge again.

The effect receipt is therefore scoped to the continuity ID and the epoch, not to the local instance. When the execution lands somewhere new with a new local instance ID, the receipts still belong to the same logical execution. A step that already dispatched is still recorded as dispatched.

The epoch in the receipt is what turns this into enforcement rather than bookkeeping. An old owner attempting to dispatch after handoff produces a `Stale` outcome at the storage boundary — distinct from `Duplicate`, because the two mean different things operationally. `Duplicate` is a workflow bug; `Stale` is a handoff race, which is a very different incident to page someone about.

The general principle: durable identity has to be the logical execution, not the physical run. Anything keyed to the local instance ID silently loses meaning the moment the execution moves, and it loses meaning quietly — the data is still there, it just no longer refers to anything the new owner will consult.

A workflow needing an S3 credential and a GPU cannot run on a phone. Discovering that after the handoff means an execution stranded on a runtime that can never advance it — and, because ownership already moved, the origin can no longer take it back without another transfer.

So the portable capsule carries its requirements, and the destination is checked against them before ownership moves rather than after:

pub struct CapsuleRequirements {
    pub handlers: Vec<String>,      // every step handler must be registered
    pub plugins: Vec<String>,
    pub credentials: Vec<String>,   // named, not the secrets themselves
    pub regions: Vec<String>,       // residency constraints
    pub hardware: Vec<String>,
    pub requires_network: bool,     // can it make progress offline?
}

pub enum RuntimeKind { Server, Edge, Mobile, Desktop, Browser }

The capsule is versioned with an explicit compatibility rule — same major version, and the reader's minor must be at least the offered minor. A destination running older code refuses a capsule it cannot fully interpret instead of silently ignoring fields it does not recognise. Dropping an unknown field during a state transfer is how you lose a constraint nobody notices was missing.

Delegation to a device adds more checks: current ownership epoch, live same-tenant registration for both source and destination, every handler the isolated sub-sequence requires, a destination-bound signed grant, and a one-time token. Shared mutable execution state is deliberately not transferred — the device receives an isolated sub-sequence, not the whole execution, so a compromised device cannot reach beyond its slice.

Once an execution has crossed three runtimes and two organisations, "what happened?" needs an answer that does not require trusting whoever is answering. That is a hash chain, with each entry committing to its predecessor:

pub struct ProvenanceEntry {
    pub continuity_id: ContinuityId,
    pub epoch: ExecutionEpoch,
    pub kind: String,
    pub redacted_summary: Option<String>,  // bounded, operator-safe
    pub payload_sha256: String,            // the payload itself is not stored
    pub previous_sha256: Option<String>,   // chain link
    pub entry_sha256: String,
    pub signing_key_id: Option<String>,
    pub signature: Option<String>,         // Ed25519
}

Storing digests rather than payloads is what makes the chain safe to keep and safe to share. A regulator or a counterparty can verify that a decision was recorded and unaltered without the record itself becoming a second copy of the customer data. Retention obligations and privacy obligations stop fighting each other.

Key rotation is handled with a trusted-key registry mapping retired key IDs to public keys, so historical entries stay verifiable after the signing key changes. The active key is trusted automatically; only retired ones need registering. Getting this wrong means every rotation invalidates your entire audit history, which teams usually discover at the worst possible moment.

Federation is where the trust assumptions genuinely change. Inside one deployment the runtimes are yours. Across a federation boundary the destination is someone else's, and the design has to assume it might be hostile or simply wrong.

Peers are explicit operator configuration — never discovered, never self-registered. Each entry requires a bounded HTTPS endpoint, a non-empty tenant allowlist, and a SHA-256 trust-root digest. The list is capped at 128 peers, and any invalid entry disables federation entirely rather than leaving a partially trusted configuration running. That last rule is the important one: a config file with one malformed peer among twenty is an ambiguous security posture, and ambiguous security postures should not boot.

An outbound envelope binds peer, tenant, continuity ID, epoch, destination, payload digest, issue time, and expiry, signed with the continuity key. Maximum lifetime is five minutes. Verification uses the configured key for that peer — never a key supplied in the request, which would make the signature self-certifying and therefore worthless.

Delivery is deliberately not automatic. Signing produces an envelope; it does not make an outbound network call. Separating "authorise this transfer" from "perform this transfer" means an API that signs cannot be turned into a request-forgery primitive, and the operator can see exactly where egress happens.

On the receiving side, a durable database receipt admits exactly one delivery per tenant/peer/message tuple, so a replayed envelope is rejected rather than processed twice. Expired receipts are pruned at ingestion — an envelope past its TTL can no longer verify, so retaining its receipt would grow the table forever to protect against something already impossible.

The single sentence that shapes this whole subsystem: the engine fails closed when it cannot prove ownership or whether an external effect happened.

Ambiguous region on a confidential operation? Refuse. Destination lacking verified signed trust? Refuse. Effect in an unknown state? Block the retry. Federation config with one bad entry? Disable federation. In every case the alternative — proceed and hope — trades a visible stall for an invisible violation, and invisible violations are found by auditors and customers rather than by engineers.

This has a real cost that should be stated rather than buried. Fail-closed systems stall, and stalls need operators. If you build this, you also have to build the queue of blocked executions, the tooling to resolve them, and the alerting on queue depth — otherwise you have swapped silent corruption for silent paralysis, which is better but not by as much as you would like.

The wider point for anyone designing distributed state transfer: most of the complexity here is not moving the data. It is proving that exactly one party has authority at any instant, that the record of what happened survives the move, and that nobody can quietly edit it afterwards. Those three properties are what separate a running execution from a file.