Snapshot vs Event Replay: Two Ways to Build a Durable Workflow Engine

A durable workflow engine has one hard requirement: after a crash, a workflow must continue from where it was rather than from the beginning. There are two ways to satisfy it, and the choice propagates into nearly every other design decision in the system.

**Event replay.** Persist an append-only log of everything that happened. On resume, re-execute the workflow function from the top, feeding it recorded results instead of calling the real thing. The function reaches the point it left off and continues. Temporal, Cadence, and Azure Durable Functions work this way.

**State snapshots.** Persist the current state after each step completes. On resume, load the state and continue from the next step. No re-execution. This is what I built.

Both work. The interesting part is that neither dominates, and the marketing on both sides tends to describe the other model's worst case as its normal case. What follows is the comparison I wish I had read before choosing.

Replay only produces the correct state if the workflow function is deterministic. Re-running it must take the same branches and make the same calls in the same order. Which means your workflow code cannot do a long list of ordinary things:

// Time — a replay hours later takes a different branch
if (Date.now() - startTime > TIMEOUT) { ... }

// Randomness — replay picks a different variant
const variant = Math.random() > 0.5 ? "a" : "b";

// Ambient state — the config changed since the original run
const limit = await redis.get("rate_limit");

// Iteration order — not guaranteed stable across runtimes
for (const key of Object.keys(payload)) { ... }

Replay engines solve this properly: they supply deterministic replacements for time and randomness, and they detect non-determinism at replay and fail loudly rather than corrupting state. It is a real solution, not a papering-over. But it is a constraint every developer on the team has to internalise, and violations are found at runtime rather than at compile time.

Snapshots have no determinism requirement because nothing is re-executed. Each step runs exactly once, its output is persisted, and the next step reads that output. `Date.now()` is fine. `Math.random()` is fine. Reading config mid-workflow is fine. The reason is not cleverness — it is that snapshots simply never re-run the code that would have to be deterministic.

Replay reconstructs state by re-executing against the full history. A workflow with 40,000 recorded events replays 40,000 events every time it wakes up. For a workflow with a handful of steps this is irrelevant. For a long-running process — a 90-day subscription lifecycle, a document approval that sits for weeks, an agent loop with thousands of tool calls — it is a growing cost paid on every single wake-up.

The standard mitigation is continue-as-new: periodically finish the workflow and start a fresh one carrying forward the state you need. It works, and it is also orchestration code your team writes and maintains that exists purely to serve the persistence model. In one production Temporal codebase I reviewed, continue-as-new plumbing, state externalisation, payload compression codecs, and search-attribute registration came to over 2,400 lines — roughly a tenth of the codebase, dedicated to working around the framework rather than solving the business problem.

Snapshot resume is a primary-key lookup. Step count does not affect it. A workflow on its four-thousandth step resumes exactly as fast as one on its second.

The cost lands elsewhere, and it is worth naming: the snapshot is the only authoritative state, so a bug that writes a bad snapshot has no history to reconstruct from. Replay engines have a genuine durability advantage here — the log is the truth and the state is derived, which means a state bug is recoverable by fixing the derivation. I compensate with an append-only audit log and periodic checkpoints, but that is a mitigation, not an equivalent.

Long-running workflows outlive the code that started them. You deploy on Tuesday; a workflow that started Monday is still running Friday. Both models have to answer what happens to it, and neither answer is comfortable.

Under replay, deploying changed workflow code means in-flight workflows will replay their history against the new function and can diverge from what was recorded — a non-determinism error. The fix is version markers in the code: an explicit branch keeping the old path alive for old executions. Correct, and it accumulates. Workflow functions acquire archaeological layers of version branches that nobody can safely delete because nobody is certain the last affected execution has finished.

Under snapshots, in-flight workflows continue on their pinned sequence version and new instances get the new one. No version markers in code. But the state schema is now the compatibility surface: change the shape of what a step writes, and older in-flight instances carry state the new steps do not understand. It is the same problem relocated — from code branches to data migration.

I put the safety net in the release path rather than in the workflow code. A semantic diff between two versions classifies each change and refuses to promote a risky one silently:

struct StepFacts {
    handler: String,           // changed handler = side-effect risk
    params: Value,
    retry: Option<Value>,      // retry policy changes are behavioural
    timeout: Option<Value>,
    compensation: Option<Value>,
    when: Option<String>,      // guard changes alter reachability
    output_schema: Option<Value>,  // schema changes break consumers
    // ... every field the DSL exposes, compared structurally
}

Paired with a static dataflow compiler that checks every `outputs.*` and `data.*` reference against declared schemas before release, and a canary gate that promotes a version only after it has run clean traffic. That is a lot of machinery — machinery replay engines partly get for free, because non-determinism detection catches a subset of these problems at runtime without anyone building a compiler.

Replay engines cap individual history events — Temporal's default is 2MB per payload with a 50MB history limit. This is not arbitrary: the whole history moves between server and worker on every replay, so unbounded payloads would make replay unbounded too. The mitigation is a codec that swaps large payloads for external references, which is a genuinely good pattern and also more infrastructure to run.

Snapshot state is a database row, so the limit is whatever your column allows. I still externalise oversized outputs to a separate table with a TTL, because a 40MB JSONB column ruins query performance for everyone — but that is a performance decision I can tune, not a hard protocol limit I must design around.

The snapshot model has a quieter advantage here: state is directly queryable. Finding every workflow stuck on a particular step is a normal SQL query against a GIN-indexed JSONB column. Under replay, the state lives inside the workflow function's memory during execution, so exposing it requires query handlers — code you write per workflow, deployed with the workflow, to answer questions you did not anticipate when you wrote it.

-- Every instance waiting on a specific approval, across all workflows
SELECT id, sequence_id, context->'data'->>'order_id'
FROM task_instances
WHERE state = 'waiting'
  AND metadata @> '{"awaiting": "legal_review"}';

-- Backlog by tenant, right now
SELECT tenant_id, count(*)
FROM task_instances
WHERE state = 'scheduled' AND next_fire_at <= now()
GROUP BY tenant_id ORDER BY 2 DESC;

During an incident this matters more than it looks on a feature comparison. The questions you need answered at 3am are rarely the ones you built a query handler for.

Temporal's production topology is a frontend, history, matching, and worker service, plus Cassandra or Postgres, plus Elasticsearch for visibility, plus your own workers. Every component is well-engineered and there for a reason — that architecture is what lets it scale to enormous workloads with strong isolation.

It is also a full-time operational job. On a team of five, adopting it means one person becomes the workflow-infrastructure person. That is the real cost, and it does not show up in a feature matrix.

My engine is one Rust binary and Postgres. Five roles — all-in-one, control, executor, gateway, edge — assemble narrower surfaces from the same binary, so scaling out is a deployment topology change rather than a new set of services. SQLite runs the identical engine for tests, embedded use, and on-device execution, which means a test suite needs no running server at all.

The honest counterpoint: Temporal has years of production hardening across thousands of deployments, a large ecosystem, and people you can hire who already know it. "One binary" is an operational advantage and an ecosystem disadvantage, and which one dominates depends entirely on your team size and risk appetite. Below roughly ten engineers, operational simplicity usually wins. Above that, the ecosystem often does.

Neither model is correct in general. Both are correct for particular shapes of problem, and the shape is usually knowable in advance.

• **Choose event replay** when you need to replay production failures against new code locally, when workflows are short enough that history stays small, when you have the operational capacity for a multi-service cluster, or when the ecosystem and hiring pool matter more than the footprint.
• **Choose snapshots** when workflows run for weeks or months, when developers need to write ordinary non-deterministic code, when state must be queryable by operators who did not write the workflow, when you need to run on-device or embedded, or when one binary plus Postgres is the operational budget you actually have.
• **Choose neither** when a cron job and an idempotent script would do. The best orchestration decision is often not to orchestrate — a durable engine earns its complexity only when steps genuinely need to survive failures, wait on humans, or coordinate across systems.

That last one deserves more weight than it usually gets. A large share of workflow-engine adoptions are solving a problem that a retry loop and a status column would have solved, and the engine becomes a dependency the team maintains forever in exchange for durability they were already getting.

What I would push back on regardless of which you choose: any claim that one model has no downsides. Replay buys determinism-based debugging and pays in constraints and history cost. Snapshots buy freedom and O(1) resume and pay in weaker reconstruction and self-built release safety. Those are the actual trades, and an architect who cannot name the cost of their own choice has not finished thinking about it.