Every automation platform eventually learns the same lesson: the network delivers some events twice, some workers crash halfway through a step, and some third-party APIs time out after they have already done the work. At-least-once delivery is the only guarantee a distributed system can honestly make. Exactly-once effects have to be built on top, one step at a time. This is how the execution engine behind Workflow Automation Studio does it, and what we changed after running it in production.
A Run Is a Log of Intentions, Not a Function Call
In the visual builder an automation looks like a flowchart: a trigger, a few conditions, a chain of actions. Underneath, we never execute that flowchart as a single process. A trigger event is first written to a durable event log in the same transaction as the change that caused it, using the transactional outbox pattern. A scheduler reads the log, resolves which automation versions subscribe to the event and creates a run record for each one. Stateless workers then pick up individual steps, execute them and write the outcome back before the next step becomes eligible.
That separation buys three properties. A worker can die at any point and another worker resumes from the last recorded step. Each run is pinned to the automation version that was live when its trigger fired, so editing an automation never changes a run already in flight. And because every step outcome is persisted, the complete history of a run is available for inspection and replay without re-deriving anything.
The cost is that every step must be safe to execute more than once. A worker can finish calling Jira, crash before recording success, and the step will run again. The rest of the design is about making that second execution harmless.
You cannot prevent duplicate execution in a distributed system. You can make a duplicate execution indistinguishable from a single one.
Deriving idempotency keys that survive retries
Each step gets an idempotency key before it executes. The key has to be identical on every attempt of the same logical step and different for anything that is genuinely a new operation. Our early implementation included the attempt timestamp in the hash, which made every retry look new and produced exactly the duplicates the key was meant to prevent. The current derivation uses only inputs that are fixed when the run is created:
import hashlib
def step_key(run, step, index=0):
# Stable across attempts: no timestamps, attempt numbers or worker ids.
# No automation version either: a replay on a newer version must still
# recognize the steps that already succeeded.
material = "|".join([
run.workspace_id,
run.automation_id,
run.trigger_event_id, # unique per source event, deduplicated at ingest
step.id,
str(index), # position inside a for-each loop
])
return hashlib.sha256(material.encode("utf-8")).hexdigest()
The trigger_event_id matters as much as the step identity. Inbound webhooks and integration events are deduplicated at ingest using the provider’s delivery id where one exists, and a hash of the normalized payload within a five-minute window where it does not. Without that, two deliveries of the same GitHub event create two runs with different run ids, and no amount of step-level idempotency will help. For runs started through POST /v1/automations/{id}/runs, the caller’s Idempotency-Key header takes the place of the event id.
The effect ledger
Keys are only useful if something remembers them. Before a step performs a side effect, the worker claims the key in an effect ledger protected by a uniqueness constraint. If the claim succeeds, the step runs and records its result. If the key already holds a completed result, the worker returns that result without calling anything.
INSERT INTO effect_ledger (idem_key, run_id, step_id, status, claim_token, claimed_until)
VALUES ($1, $2, $3, 'in_progress', $4, now() + interval '2 minutes')
ON CONFLICT (idem_key) DO UPDATE
SET claim_token = EXCLUDED.claim_token,
claimed_until = EXCLUDED.claimed_until
WHERE effect_ledger.status = 'in_progress'
AND effect_ledger.claimed_until < now()
RETURNING claim_token;
The claimed_until lease handles crashed workers. A returned row means this worker owns the step: either the key is new, or an earlier claim outlived its lease and has been taken over. No row means the key is held by a live claim or has already completed, so the worker reads the entry and either returns the stored result or backs off. The completion write is conditional on the worker’s claim_token, so a worker that stalls past its lease and wakes up later cannot overwrite the result recorded by the worker that replaced it. Completed entries are kept well beyond the longest retry schedule and the replay window for dead-lettered runs, so a late duplicate always finds its answer.
Not every API can be made idempotent
The ledger protects our side of the boundary. Third-party APIs vary, so each connector action declares one of three delivery classes:
- Native. The provider accepts an idempotency key. We pass our step key through and the provider guarantees the rest.
- Reconcilable. The provider does not accept keys, but created objects can be tagged and searched. Before retrying an ambiguous failure, the connector looks for an object carrying the step key in a custom field or metadata, and adopts it if found.
- Unsafe. Neither is possible, as with some email and SMS gateways. Ambiguous failures, such as a timeout after the request was sent, are not retried automatically. The run pauses and raises a failure alert with the exact request, so a person decides whether to resend.
Declaring the class explicitly turned out to be as important as implementing it. It forces every connector author to answer “what happens if this times out after succeeding?” during code review instead of during an incident.
Retries that do not stampede
Retryable failures back off exponentially with full jitter, up to five retries. The policy is intentionally boring and identical across connectors, with two rules layered on top: a Retry-After header always wins over our schedule, and a failure classified as terminal skips the schedule and goes straight to the dead-letter queue.
retry:
max_retries: 5
base_delay: 10s
max_delay: 5m
jitter: full # retry n (1-5) sleeps random(0, min(max_delay, base_delay * 2^n))
honor_retry_after: true
retryable: [timeout, connection_reset, http_429, http_5xx, schema_violation]
terminal: [http_400, http_401, http_403, http_404, http_422]
on_exhausted: dead_letter
With these values the ceilings are 20, 40, 80 and 160 seconds, then 5 minutes, so unless a provider sends a longer Retry-After, a step waits at most ten minutes in total before its run is dead-lettered. Jitter matters at scale. When a large integration provider has a regional outage, thousands of steps fail within the same few seconds. Deterministic backoff would retry them in synchronized waves and prolong the outage for everyone; full jitter spreads the retries across the window. We also cap concurrent in-flight calls per connector per workspace, so one workspace’s backlog cannot consume another’s share of a provider’s rate limit.
After the fifth retry, the run moves to a dead-letter queue with its full context. Replaying from the dead-letter queue reuses the original run id and idempotency keys, so steps that already succeeded are skipped and execution resumes at the failed step. Admins can replay on the original automation version, which is the default, or on the latest version after fixing a field mapping.


AI steps are non-deterministic, so record them
AI steps introduce a problem that ordinary API calls do not have: the same input can produce a different output on a second attempt. If a run drafts a customer update, crashes and retries, a naive retry sends a subtly different message from the one a reviewer approved.
We treat a validated AI output as a side effect in its own right. The output is stored in the effect ledger under the step key, together with the prompt template version and the model route chosen by the router. Retries and replays read the recorded output instead of generating a new one. Generation reruns only when someone explicitly chooses to regenerate during a replay.
Every AI step also declares a JSON Schema for its output, and nothing reaches the next step until the output validates. Before the step is marked failed, an invalid response gets one repair attempt, in which the validator errors are sent back with the original request. If the repaired output is still invalid, the step fails with schema_violation and follows the same retry path as any other failure; because only validated outputs are written to the ledger, each retry generates afresh. In our own workspace, roughly 1 in 400 extraction steps needs the repair pass, and about 1 in 20,000 exhausts its retries and reaches the dead-letter queue, almost always because the source document was truncated or empty.
{
"type": "object",
"required": ["category", "severity", "customer_facing"],
"additionalProperties": false,
"properties": {
"category": { "enum": ["bug", "feature_request", "question", "incident"] },
"severity": { "enum": ["low", "medium", "high", "critical"] },
"customer_facing": { "type": "boolean" },
"summary": { "type": "string", "maxLength": 280 }
}
}
Observability you can act on
Every run produces a trace: one span per step, with queue wait, execution time, attempt count, delivery class and the idempotency outcome (executed, reused or reconciled). The run log in the builder is a rendering of that trace, so the view an admin sees is the same data our on-call engineers use. Payload fields are redacted according to the workspace’s PII settings before logs are written, not at display time.
Alerting took the longest to get right. Alerting on individual step failures produced noise within a week, because transient 5xx responses are normal and retries absorb them. Failure alerts now fire on conditions that need a human: a run reaching the dead-letter queue, a connector’s retry rate over a 15-minute window crossing three times its seven-day baseline, or a paused run on an unsafe delivery class. A dead-lettered run also emits the automation.run.failed webhook, so teams can route it into their own incident tooling.
Five things production taught us
- Deduplicate at ingest. Step-level idempotency cannot fix two runs created from one event.
- Keep attempt numbers, timestamps, worker ids and automation versions out of idempotency keys.
- Make every connector declare what happens when a request times out after succeeding.
- Record AI outputs as effects, so retries and replays repeat what a reviewer saw.
- Alert on dead letters and rate changes, not on single failures that retries will absorb.
Build on the same engine
Idempotency keys, jittered retries, the dead-letter queue, replay, versioned automations and per-run logs apply to every automation built in Workflow Automation Studio, which is included in the Business and Enterprise plans (see pricing). Runs are pooled per workspace: 25,000 a month on Business and 100,000 on Enterprise. Triggers, AI step types and connector details are covered on the Workflow Automation Studio page.