Exactly-Once Is a Lie. Here Is What a Workflow Engine Can Actually Promise

A workflow step calls a payment API. The engine crashes between sending the request and persisting the response. On restart it finds a step that started and never finished. Retrying might charge the customer twice. Not retrying might leave the order unpaid forever.

This is not a rare edge case; it is the defining problem of durable execution. Every framework that advertises "exactly-once" is describing exactly-once *state transitions inside its own database* — which is genuinely valuable and completely different from exactly-once *effects on someone else's system*. Those are separated by a network, and no protocol closes that gap without cooperation from the other side.

The honest decomposition has three parts. Internal state transitions can be exactly-once, because one database transaction owns them. External effects can be at-most-once, because you can durably record intent before dispatching. And the outcome of a dispatched effect is sometimes genuinely unknown — at which point the only correct behaviour is to stop and say so.

The usual model for a side effect is a flag: done or not done. That model has no way to represent "we sent it and never heard back," which is the state that actually causes incidents. So effects get a full lifecycle with eight states:

pub enum EffectState {
    Planned,      // decided to do it
    Prepared,     // receipt durably written, not yet sent
    Dispatched,   // sent; outcome not yet known
    Committed,    // provider confirmed
    Unknown,      // sent, no answer — blocks automatic retry
    Verified,     // reconciled against the provider after the fact
    Compensated,  // undone by a compensating action
    Abandoned,    // explicitly written off by an operator
}

The important state is `Unknown`, and it exists so the engine never has to guess. `Dispatched` becoming `Unknown` on restart is the crash case: the process died after sending and before recording an outcome. `Verified` is how it gets out — a reconciliation job or an operator queries the provider, discovers what actually happened, and settles the receipt with evidence.

Every transition is a compare-and-swap against the stored state, not a blind write. If the CAS fails, some other process already advanced the receipt, and the current caller is working from a stale view — so it stops rather than racing:

async fn advance(&mut self, next: EffectState) -> Result<(), EngineError> {
    let expected = self.receipt.state;
    let mut updated = self.receipt.clone();
    updated.transition(next, Utc::now())?;   // rejects illegal transitions

    if !self.storage
        .cas_effect_receipt(&updated.tenant_id, updated.id, expected, &updated)
        .await?
    {
        let current = self.storage
            .get_effect_receipt(&updated.tenant_id, updated.id).await?
            .unwrap_or(self.receipt.clone());
        return Err(blocked(&current));   // stop; do not re-dispatch
    }
    self.receipt = updated;
    Ok(())
}

A retry has to find the receipt the first attempt created, or the whole scheme collapses into a fresh receipt per attempt and no protection at all. Generating a random ID at dispatch time does exactly that. So the effect ID is derived, not generated:

let id = deterministic_effect_id(
    execution.continuity_id,   // which logical execution
    execution.epoch,           // which ownership generation
    instance_id,               // which run
    block_id,                  // which step
    attempt,                   // which attempt
);

Two processes racing on the same step compute the same ID and converge on one row. `ensure_effect_receipt` is an idempotent upsert, and its result is checked against the candidate — if a receipt with that ID exists but carries different evidence, that is a hash collision or an implementation bug, and the engine refuses to continue rather than reusing someone else's receipt.

Including `attempt` in the derivation is a deliberate choice with a trade-off. It means a legitimate retry after a *known* failure gets a fresh receipt and is allowed to proceed. It also means the deduplication boundary is one attempt wide, and the cross-attempt protection has to come from the `Unknown`-blocks-retry rule rather than from the ID alone. Those two mechanisms are complementary; neither is sufficient by itself.

The receipt also captures a destination fingerprint, a SHA-256 of the canonicalised request, any caller-supplied idempotency key, and the provider's own receipt ID once it comes back. That is the evidence a reconciliation job needs later, and it is why the receipt is written *before* dispatch rather than after.

Wrapping every step in a durable receipt would add two round trips to work that cannot hurt anyone. A `log` step or a `transform` has no external footprint. So the guard is only constructed for handlers that can touch the outside world — and the classification is deliberately pessimistic:

const SIDE_EFFECT_BUILTINS: &[&str] = &[
    "http_request", "llm_call", "tool_call", "mcp_call", "agent",
    "emit_event", "send_signal", "self_modify",
    "memory_store", "memory_delete", "blob_put", "embed",
];

pub(crate) fn handler_has_side_effects(handler: &str) -> bool {
    SIDE_EFFECT_BUILTINS.contains(&handler)
        || !BUILTIN_HANDLER_NAMES.contains(&handler)   // user handler: assume yes
}

That second clause is the one that matters. A handler the engine does not recognise is a handler someone wrote, and the engine cannot prove it does not send email. Defaulting to "protected" costs a little latency on custom steps. Defaulting to "safe" would silently exclude exactly the code most likely to have effects the engine has never seen.

This same list drives release-risk classification: changing a step that touches one of these handlers is flagged as a side-effect risk in the semantic diff between two workflow versions, not merely as a behavioural change. One classification, two consumers.

Application-level checks cannot prevent two processes from dispatching concurrently — the check and the act are separated in time. So when a workflow declares an at-most-once invariant for a class of effect, the transition to `Dispatched` becomes a conditional storage operation that returns which of three things happened:

match storage.dispatch_effect_receipt_at_most_once(&tenant, &updated).await? {
    EffectDispatchOutcome::Dispatched => { /* we own it; proceed */ }

    EffectDispatchOutcome::Duplicate => {
        record_effect_guard_results(storage, &guards, &updated, true).await?;
        Err(EngineError::InvariantViolation {          // a second dispatch was attempted
            invariant_id: guards[0].id,
            block_id: updated.block_id,
        })
    }

    EffectDispatchOutcome::Stale => {                  // an older epoch tried to act
        let current = storage.get_effect_receipt(&tenant, updated.id).await?;
        Err(blocked(&current))
    }
}

`Duplicate` and `Stale` are separate outcomes on purpose. `Duplicate` means the invariant caught a genuine double-dispatch attempt and the violation is recorded against the invariant that declared it — an operator sees which rule fired and where. `Stale` means the caller belongs to a superseded generation of the execution, which is what happens after ownership moves to another node. Collapsing them into one error would make those two very different incidents look identical in the logs.

The invariants themselves are per-workflow-version configuration rather than global engine behaviour. A workflow declares which effect kinds are at-most-once and which of those act as commit guards; the guard loads only the invariants matching the current receipt's kind. Charging a card and appending to a log do not need the same protection, and paying for the strong version everywhere would be a tax on the steps that do not need it.

Every side-effecting step pays for a durable receipt write before dispatch and a CAS after it. That is real latency on the hot path, and it is the price of being able to answer "did it happen?" after a crash. There is no version of this that is free.

It also means workflows can get stuck in a way that a naive engine would not. An `Unknown` receipt blocks automatic retry until something resolves it. If you do not build the reconciliation path — a job that queries providers and settles receipts — you have traded silent duplicate charges for a queue of manually stuck workflows. That is a better failure mode, but only if someone is watching the queue.

Which is the practical takeaway: pass an idempotency key to every provider that accepts one, and let the engine carry it in the receipt so a reconciliation can use it. The engine's job is to make sure you never dispatch twice without knowing, and to remember enough to find out what happened. The provider's job is to make a duplicate harmless if you do.

You do not need a workflow engine to apply the pattern. The parts that matter transfer to any system that calls someone else's API from inside a retryable unit of work:

1. Write intent durably before you dispatch. If the record is not committed, you cannot later prove you tried.
2. Derive the record identity from position in the work, not from a fresh UUID at call time. Retries must converge on one row.
3. Give ambiguity its own state. "Sent, outcome unknown" is not a failure and not a success, and modelling it as either causes a different bug.
4. Make the state transition conditional at the storage layer. Check-then-act in application code is not mutual exclusion.
5. Classify handlers pessimistically. Anything you did not write is side-effecting until proven otherwise.
6. Build the reconciliation path before you need it, and alert on the depth of the unresolved queue.

And say what you actually guarantee. A system that says "at-most-once, with an explicit unknown state and a reconciliation path" is more trustworthy than one that says "exactly-once" and quietly means something narrower. Engineers who have operated distributed systems know the difference, and the second answer costs you their confidence at exactly the moment you were trying to earn it.