Run run-7f3a is working on submission SUB-1004. Step 2 has just finished: the outside provider answered after eleven seconds that the roof is 22 years old, the answer is stored as evidence EV-ROOF-0412, and the worker has saved checkpoint cp-run-7f3a-0002 with state_version 2. Then the worker process dies. A deploy, an out-of-memory kill, a lost machine: the reason does not matter.
A new worker picks the run up a minute later. The previous page built the machinery that lets it know where the run stopped. This page answers what it should do next, and the answer has two halves. It should not repeat what is finished: the provider call costs money and time, and its answer is already stored. And for the steps it must run, any step that may have run before must be safe to run again, because a worker cannot always tell whether a step finished before the crash.
This is case UW-M01, the anchor of this site's own cases. The second half is case UW-M04. The agent here is a scripted stand-in, so the results on this page show how the mechanism works, not how good any model is.
Resume: the same run, the next attempt#
To resume is to continue the same run from its last checkpoint. For example, the new worker reads checkpoint cp-run-7f3a-0002, sees that the next step is search_guidance, and starts there with the roof assessment already in the state.
Each process-level try of a run is an attempt. Attempt 1 of run-7f3a ran steps 1 and 2 and crashed. The resume is attempt 2. The run_id stays run-7f3a; the attempt_number becomes 2. On this site each attempt also gets its own trace_id, because a trace records what one process did at one time, and attempt 2 is a different process at a later time. The run_id is what joins the two traces. Run identity is the home of these ids.
A resume is not a restart. A restart would begin a new run with a new run_id from step 1, and it would call the provider again. A resume is also not a replay: replay rebuilds state from recorded events without doing anything, while a resume goes on to do the remaining work.
Here is the example's runner on UW-M01. It shows that the worker died after step 2, that attempt 2 started from state_version 2, and that the provider was called once in total. A passing run proves that the resume logic in workflow.py skips completed steps in this scripted case; it does not prove anything about a real provider or a real model.
python run_memory.py --case UW-M01
Memory run · UW-M01 · agent: scripted stand-in (no model called) · store out/memory.sqlite
UW-M01 The worker dies right after the roof assessment was fetched and checkpointed
run run-7f3a · thread-SUB-1004 · corr-5c1e · SUB-1004 (BRK-017)
attempt 1 trace 1d237d5c… crashed from the start steps run: load_submission, get_roof_assessment
attempt 2 trace f1cb96bc… paused from state_version 2 steps run: search_guidance, find_similar_decisions, check_eligibility, write_recommendation, record_recommendation, notify_cockpit
provider calls 1 · recommendation rows 1 · escalate citing R-ROOF-03 · run paused at underwriter_review
expected (built on UW-004): PASS · 11 of 11 checks
1 of 1 cases as expected · tables: runs 1, attempts 2, checkpoints 9, events 25, evidence 1, recommendations 1, outbox 1, inbox 1, decisions 0, memories 10, shared_records 0
wrote out/report.json and out/memory.sqlite
Read the two attempt lines. Two attempts, two traces, one run. Attempt 2 ran six steps and none of the first two. The run did not complete: it paused at underwriter_review, because a person decides. The model recommends; a person decides.
The function that decides where an attempt starts is short. It marks a previous attempt that is still running as crashed, numbers the new attempt, and loads the newest checkpoint. This excerpt is from workflow.py, function _begin_attempt:
number = prev[0]["attempt_number"] + 1 if prev else 1
cp = latest_checkpoint(conn, run_id) if settings.checkpoints else None
state = json.loads(cp["state_json"]) if cp else {}
next_step = cp["next_step"] if cp else STEP_NAMES[0]
The gap between doing a step and saving that it was done#
The anchor case was the easy one. The crash came after the checkpoint of step 2 was saved, so the new worker knew exactly where to start.
Now move the crash. In case UW-M04, run run-1e6b for SUB-1001 reaches step 7, record_recommendation. The write commits a recommendations row: a decline citing R-ROOF-01. Then, before the worker saves the checkpoint for step 7, it dies.
The newest checkpoint says step 6 is done and step 7 is next. So attempt 2 runs step 7 again. It has no way to know that the row is already there: the only record that step 7 finished would have been the checkpoint it never saved.
This gap exists in every system that does work and then records that it did it. No ordering removes it. If you save the checkpoint first and crash before the work, the record says done when it is not. If you do the work first and crash before the checkpoint, the work happens twice. Workflow systems choose the second, because a missed step is worse than a repeated one. That choice has a name: at-least-once execution, meaning every step runs one or more times, never zero.
Production pitfall. At-least-once is not a bug to be fixed with better crash handling. It is a property of doing work in one place and recording it in another. Any step with a side effect, a change outside the run's own state such as a provider call, a stored row or a sent message, must be designed to tolerate running twice.
Idempotency keys: making the repeat harmless#
Idempotent means doing something twice leaves the same result as doing it once. For example, "set the roof age to 22" is idempotent; "add a recommendation row" is not.
The write in step 7 is an add, so the example makes it idempotent with a key. An idempotency key is a value that names one intended effect; the store keeps it with the result, and a repeat with the same key returns the first result instead of acting again. The tool record_recommendation takes the key as an argument. The example builds it like this, in idempotency.py, function idempotency_key:
def idempotency_key(run_id: str, step: str, attempt_number: int, mode: str) -> str | None:
"""The key for one write.
run_step run-7f3a:record_recommendation (right: one key per intended effect)
attempt run-7f3a:2:record_recommendation (wrong: a resumed attempt gets a new key)
none no key (every delivery stores a row)
"""
if mode == "none":
return None
if mode == "attempt":
return f"{run_id}:{attempt_number}:{step}"
return f"{run_id}:{step}"
The three modes are there to compare. Only run_step is the design. The runner on UW-M04 stores the recommendation under each mode and counts the rows:
UW-M04 The worker dies after record_recommendation committed, before its checkpoint was saved
run run-1e6b · thread-SUB-1001 · corr-a210 · SUB-1001 (BRK-011)
attempt 1 trace 12509e90… crashed from the start steps run: load_submission, get_roof_assessment, search_guidance, find_similar_decisions, check_eligibility, write_recommendation, record_recommendation
attempt 2 trace 284fe3f8… paused from state_version 6 steps run: record_recommendation, notify_cockpit
provider calls 1 · recommendation rows {'run_step': 1, 'none': 2, 'attempt': 2} · decline citing R-ROOF-01 · run paused at underwriter_review
expected (built on UW-001): PASS · 8 of 8 checks
record_recommendation appears in both attempts: the repeat happened, as at-least-once says it will. With the key per run and step, the second call found the key and returned "already stored, returned the first result": one row. Without a key, two rows. With a key that includes the attempt number, also two rows, because attempt 2 built a different key, run-1e6b:2:record_recommendation, which the store had never seen.
That last result is the mistake this page exists to prevent. A key per attempt looks careful, and it fails exactly in the case keys are for. The key must name the intended effect, "the recommendation of this run", and that effect does not change when the attempt does.
Try the three choices in the lab. Set how many times the write arrives and which key is used, and see the rows the underwriter would find.
| Key | Arrives once | Arrives twice (crash before checkpoint) | Arrives three times |
|---|---|---|---|
| None | 1 row | 2 rows | 3 rows |
Per attempt (run:attempt:step) | 1 row | 2 rows | 3 rows, if each arrival is a new attempt |
Per run and step (run:step) | 1 row | 1 row | 1 row |
Two rows mean the underwriter cockpit lists two recommendations for one submission. Here both say decline, which is confusing. In a case where the stand-in, or a model, answers differently on the second try, the two rows disagree, and the underwriter has no way to know which one the evidence supports.
The whole picture of the resume#
The lab below puts both cases together. Choose the step after which the worker dies, whether the checkpoint of that step was saved before the crash, and whether checkpoints are kept at all. It shows the ten steps of attempt 1 and attempt 2 side by side, and counts provider calls and recommendation rows.
| Step | Attempt 1 | Attempt 2 (UW-M01) |
|---|---|---|
1 load_submission | ran | skipped: read from the checkpoint |
2 get_roof_assessment | ran (provider call 1) | skipped: read from the checkpoint |
3 search_guidance | (crashed before) | ran |
4 find_similar_decisions | ran | |
5 check_eligibility | ran | |
6 write_recommendation | ran | |
7 record_recommendation | ran: 1 row | |
8 notify_cockpit | ran | |
9 underwriter_review | paused for the underwriter | |
| Provider calls, recommendation rows | 1 call, 1 row |
The diagram shows the same run as a timeline, with the crash in the middle.
Diagram: UW-M01 as a timeline · two attempts, two traces, one run
Time runs down the page. Attempt 1 runs steps 1 and 2 and saves a checkpoint after each, then the worker dies. Attempt 2 starts with a new trace, loads state version 2 and continues at step 3. The write in step 7 carries the key run-7f3a:record_recommendation, so a repeat would store nothing new.
- Worker (attempt 1) to Provider (stand-in): get_roof_assessment(SUB-1004). Provider call 1. The answer is a roof age of 22 years.
- Worker (attempt 1) to The store: save evidence EV-ROOF-0412 and checkpoint v2. The checkpoint after step 2: roof_assessed, next search_guidance.
- Worker (attempt 1): worker dies. Attempt 1 ends without finishing. Its status becomes crashed when attempt 2 starts.
- Worker (attempt 2) to The store: load latest checkpoint: v2, next step 3. Resume: same run_id, attempt_number 2, new trace_id.
- Worker (attempt 2) to The store: steps 3 to 6, a checkpoint after each. No provider call: the roof assessment is in the loaded state.
- Worker (attempt 2) to The store: record_recommendation, key run-7f3a:record_recommendation. One row: escalate citing R-ROOF-03. A repeat with the same key would return the first result.
- Worker (attempt 2): pause at underwriter_review. The run waits for a person. The model recommends; a person decides.
Read it top to bottom. Attempt 1 runs steps 1 and 2 and saves a checkpoint after each. The worker dies. Attempt 2 starts with a new trace, loads state_version 2, and continues at step 3. Step 7 writes with the key run-7f3a:record_recommendation. Had the crash come between step 7's write and its checkpoint, the repeat in attempt 2 would have met the key and stored nothing new.
A retry is not an attempt#
One more distinction keeps the vocabulary straight. Case UW-M03: run run-c4d0 for SUB-1006 calls the provider, which times out. The worker is alive and well. It waits half a second and calls again. That is a retry policy: rules for trying one call again inside a step, such as how many tries and how long to wait between them.
UW-M03 The provider times out; the retry policy tries again inside the same attempt, then falls back
run run-c4d0 · thread-SUB-1006 · corr-0d83 · SUB-1006 (BRK-011)
attempt 1 trace e16825c3… paused from the start steps run: load_submission, get_roof_assessment, search_guidance, find_similar_decisions, check_eligibility, write_recommendation, record_recommendation, notify_cockpit
provider calls 2 · recommendation rows 1 · request_info citing no rule · run paused at underwriter_review
expected (built on UW-006): PASS · 6 of 6 checks
There is one attempt. There are two provider calls, recorded as two call attempts inside step 2. When both time out, the step falls back: the recommendation is request_info with the roof assessment listed as missing. The attempt_number never moved, because the process never died.
| Call retry | New attempt | |
|---|---|---|
| Triggered by | A failed call (timeout, error) | A dead worker |
| Who decides | The retry policy, inside the step | The workflow runtime, on restart |
attempt_number | Unchanged | Increases by one |
trace_id | Unchanged (same trace, more spans) | New |
| Starts from | The same step, same state in memory | The last checkpoint |
| In the example | UW-M03: 2 call attempts in attempt 1 | UW-M01: attempt 2 from state_version 2 |
Both can repeat a side effect, and both need the key. A provider call that timed out on the client may still have been processed by the provider. If that call had a cost or an effect, it would need its own idempotency key, passed to the provider if the provider accepts one.
What this is not. How many retries, how long to wait, when to stop calling a struggling provider (a circuit breaker), and what the service level is, belong to reliability engineering: reliable.sammuthu.com. This site covers only what must be stored so that a repeat is safe.
Exactly-once is at-least-once plus deduplication#
People often ask for exactly-once execution. Look at what the example actually does. Step 7 ran twice in UW-M04. That is at-least-once, and it cannot be avoided. What the underwriter sees, one row, is the effect of exactly-once. The difference was made by the key and the store's check of it.
Exactly-once is therefore not a property of execution. It is a property of the effect, produced by running at least once and dropping repeats. Exactly-once is at-least-once plus deduplication. Any system that promises exactly-once is doing the deduplication somewhere; the useful question is where, and over what window.
For a backend engineer. You have met this in Kafka consumers and payment APIs. A consumer that commits its offset after processing will process a message twice after a crash; the consumer must deduplicate, by a message id or by a natural key, or make its write an upsert. HTTP APIs that take payments or orders often accept a client-generated key for the same reason; the IETF has an active draft, "The Idempotency-Key HTTP Header Field", that standardizes such a header for retried POST and PATCH requests. The insurance agent is one more client with the same problem, and a slower one: minutes pass between a write and its checkpoint, not milliseconds.
The failure, shown#
Without checkpoints, UW-M01 restarts from step 1: two provider calls, and the roof assessment is fetched again. Without the key, UW-M04 leaves two recommendation rows. With a key per attempt, UW-M04 still leaves two rows, which is the failure most likely to pass review, because the code visibly has a key.
The Evaluation page turns this into a test that runs every crash point; its grid shows which combinations of checkpoint and key survive which crash.
Framework implementation#
Durable workflow engines and agent frameworks resume runs for you, and some document that a step may run more than once and must be idempotent. How each named engine resumes, and what it says about repeated steps, is on Frameworks. None of them can choose your idempotency key: only your code knows what one intended effect is.
A checklist for Monday#
- Kill the worker after each step of a run, in a test. Does the next attempt start from the last checkpoint?
- List every step with a side effect. Which of them can run twice today without harm?
- For each write, what is the idempotency key? Does it include the attempt number, a timestamp or a random value? If so, it is wrong.
- Where does the store check the key, and is the check in the same transaction as the write?
- Are call retries and new attempts recorded separately, so that a trace or a dashboard can tell them apart?
Advanced: choosing the scope and lifetime of a key
The key names one intended effect. In the example that is "the recommendation of run run-7f3a", so the key is the run and the step. Other effects need other scopes. A message to the cockpit about a recommendation is one effect per recommendation, so its key can derive from the recommendation's key. A provider call whose result is cached per submission might use the submission and the provider's request type. A step that runs in a loop needs the loop index in its key, or the second iteration will be swallowed as a repeat of the first.
The key must live as long as a repeat can arrive. For a resume inside one run, that is the life of the run. For messages that a broker can redeliver days later, it is longer. Expiring keys too early turns a late repeat into a duplicate; never expiring them grows the table without bound. A unique index on the key column is the cheapest correct implementation, because it makes the check and the write one atomic operation.
There is a subtle case: the second call arrives with the same key and a different body. In the example the stand-in agent is deterministic, but with a model the recommendation on attempt 2 may differ. Returning the first result is the right answer for the underwriter, who should see the recommendation the evidence was reviewed against. Logging the difference is worth doing, because it tells you how often the model's answer moves on identical input.
Advanced: who notices that a worker died
The example marks an attempt as crashed when the next attempt starts and finds it still running. In production something must start that next attempt. Common designs are a lease (the worker holds a lock with an expiry and renews it; when it stops renewing, another worker takes the run), a heartbeat table scanned by a supervisor, or a queue that redelivers an unacknowledged task. All three can declare a slow worker dead while it is still working, which produces two live attempts on one run. The defenses are the same as for the crash: idempotent effects, and a fencing token, such as the attempt number written with each checkpoint, so that a checkpoint from a superseded attempt is rejected. Which runtime notices, and when, is a question for orchestrate.sammuthu.com; what must be stored so that two attempts cannot both win is the question of this page.
In one breath#
A resume continues the same run from its last checkpoint as the next attempt, with a new trace and the same run_id. Because a crash can fall between doing a step and saving that it was done, steps run at least once, and every step with a side effect must be idempotent. The idempotency key names the intended effect, so it is built from the run and the step, never the attempt. Exactly-once is at-least-once plus deduplication.
The same idea for a coding agent: a session that resumes from its journal should find that a migration already ran, and a migration script that checks whether its change is already applied is the coding agent's idempotency key.
The recommendation row is now stored once. But the run also has to tell the underwriter cockpit about it, and a row and a message cannot be committed in one step. The transactional outbox.