Durable Workflow Engine Architecture
Beyond Temporal — Snapshot-Based Execution
Updated
Durable workflow engines let you write multi-step processes that survive crashes, restarts, and deployments. The dominant approach — history replay, pioneered by Temporal — works but imposes a tax on every developer who touches workflow code: determinism constraints, payload size limits, history bloat, and activity ceremony. I solo-built an alternative in Rust — every crate, every SDK, every integration written by one person — that uses snapshot-based state persistence instead. No replay. No determinism constraints. O(1) resume from the last checkpoint.
What This Means for Your Business
Teams adopt Temporal because they need durable execution. Then they spend months building workarounds for its constraints. History replay means every workflow wake-up replays the entire event log — fine for short workflows, operationally expensive for long-running ones that hit the 50K event limit. Continue-as-new orchestration, state externalization, payload compression codecs, custom search attribute registration — I have measured over 2,400 lines of workaround code in a production Temporal codebase. That is 10 percent of the entire codebase dedicated to fighting the framework.
The snapshot approach is simpler: persist the current state to a Postgres row after each step. On resume, load the row and continue. No replay, no history limits, no determinism constraints. Developers write normal code — Date.now(), Math.random(), external API calls — all work without patched() wrappers or version markers. The engine handles retries, timeouts, parallel execution, and structured concurrency. The developer handles the business logic.
History Replay vs Snapshot Persistence
The two architectures solve the same problem — workflows that survive crashes — with very different costs for the teams that build on them.
| History replay (Temporal) | Snapshot-based (Orch8) | |
|---|---|---|
| Resume after crash | Replays the entire event history — O(N) per wake-up | Loads the last snapshot — O(1), no replay |
| Determinism constraints | Required: patched() wrappers and version markers; no plain Date.now() or Math.random() | None — normal code and normal libraries work |
| Long-running workflows | 50K event limit forces continue-as-new orchestration | No event limit — snapshots stay constant-size |
| Inspecting workflow state | Signal and query handlers, custom search attributes | Standard JSONB queries against Postgres |
| Large payloads | Size limits push teams to externalize state and add compression codecs | State persisted as JSONB after each step |
| Local testing | Requires a running Temporal server | Embedded SQLite mode with identical behavior |
Measured in one production Temporal codebase: 2,400+ lines — roughly 10% of the code — existed only to work around the framework's replay constraints.
How I Have Used This in Production
10-Crate Workspace Architecture
Designed a Rust workspace with clean separation across 10 crates: orch8-types (shared domain types), orch8-storage (Postgres + SQLite via SQLx), orch8-engine (core execution logic), orch8-api (Axum REST with OpenAPI via utoipa), orch8-grpc (Tonic gRPC interface), orch8-server (runtime composition), orch8-cli (Clap-based CLI), orch8-publisher (event publishing), orch8-push (push notifications), orch8-mobile (mobile API layer). Each crate compiles independently. The engine crate has zero knowledge of HTTP or gRPC — it operates on trait abstractions that the server crate wires together.
Snapshot-Based State Persistence
Implemented O(1) resume by persisting workflow state as JSONB in Postgres after each step completion. No event log to replay. State is directly queryable — GET /instances/{id}/state returns the full context without signal handlers or query boilerplate. Filter instances by any field via standard Postgres JSONB queries. Embedded SQLite mode for testing with identical behavior.
Multi-Language SDK Design
Built Node.js and Python SDKs that communicate with the Rust engine via gRPC. Steps are plain functions — no activity ceremony, no central registry, no manual error classification. The SDK handles connection management, retry coordination, and heartbeating. Workers register step handlers by convention and the engine dispatches work with per-resource rate limiting and configurable concurrency.
Technologies
Related Expertise
The workflow engine is built in Rust because durability demands correctness guarantees that garbage-collected languages cannot provide at the systems level. See why Rust for mission-critical infrastructure.
Rust for Mission-Critical Systems — Why Rust When Failure Is Not an OptionWorkflow engines generate events that downstream systems consume in real time. See how I build the streaming infrastructure that connects orchestration to live dashboards.
Real-Time Systems — WebSockets, Message Queues, and Live DataThe storage layer under a workflow engine must handle high-frequency writes and analytical reads. See how I design data layers for systems that cannot lose state.
Database Architecture and Performance — Designing Data Systems That ScaleFrequently Asked Questions
What is a durable workflow engine and why does it matter?
A durable workflow engine lets you write multi-step processes that survive crashes, restarts, and deployments — the engine persists progress so work resumes instead of being lost. The dominant approach, history replay pioneered by Temporal, works but taxes every developer with determinism constraints, payload size limits, history bloat, and activity ceremony. The alternative is snapshot-based persistence: save current state after each step and resume in O(1) from the last checkpoint, with no replay and no determinism rules.
When should a team build on Temporal versus a snapshot-based engine?
Temporal's history replay is fine for short workflows but operationally expensive for long-running ones that hit the 50K event limit, forcing continue-as-new orchestration, state externalization, and payload compression codecs. Oleksii measured over 2,400 lines of workaround code in one production Temporal codebase — 10 percent of the codebase fighting the framework. A snapshot-based engine suits teams that want developers writing normal code, with Date.now(), Math.random(), and external API calls working without patched() wrappers or version markers.
What production experience does Oleksii Vasylenko have building workflow engines?
Oleksii solo-built Orch8, a complete durable workflow engine in Rust — every crate, SDK, and integration written by one person. The 10-crate workspace separates types, storage (Postgres and SQLite via SQLx), core engine, Axum REST API with OpenAPI, Tonic gRPC, server runtime, CLI, event publisher, push notifications, and mobile API layer. He also built Node.js and Python SDKs that communicate with the engine over gRPC, with per-resource rate limiting and configurable concurrency.
What goes wrong when teams adopt durable execution frameworks?
Teams adopt durable execution for reliability, then spend months building workarounds for the framework's constraints. With history replay, every workflow wake-up replays the entire event log, and long-running workflows hit event limits that force continue-as-new orchestration, state externalization, and custom search attribute registration. Determinism constraints also infect ordinary code: developers must wrap nondeterministic calls and manage version markers. The result Oleksii measured was 2,400-plus lines of framework-fighting code in a single production codebase.
How can a team get help moving off Temporal or fixing workflow architecture?
Oleksii has built the alternative architecture and measured the difference against Temporal in production code. His snapshot-based design persists workflow state as JSONB in Postgres after each step — O(1) resume, directly queryable state via standard JSONB queries, no signal handlers or query boilerplate, and an embedded SQLite mode for testing with identical behavior. Teams spending more time fighting their workflow framework than writing business logic can contact him through ovasylenko.com.
Further Reading
Drowning in Temporal workarounds?
History replay, continue-as-new, payload compression, determinism constraints — if your team spends more time fighting the framework than writing business logic, there is a better architecture. I have built the alternative and I have measured the difference. If you need durable execution without the operational tax, let’s talk.
Discuss your workflow architecture