The worker running run-7f3a for SUB-1004 has finished two steps. It loaded the submission, and it called the outside provider, which said the roof is 22 years old. At that moment the process dies. The previous page gave the run a name that outlives the process. A name is not enough, though. The next worker needs to know three things: which step comes next, what the run has gathered so far, and which of the finished steps did something outside the run that must not happen twice.
This page builds the record that answers those three questions. It starts with the shape of the run itself, then compares two ways to save it, then draws the line that matters most: what may be computed again, and what must be read back. The case is UW-M01, the anchor of this site's own cases; the pause for a person is UW-M06.
The agent here is a scripted stand-in, so the results on this page show how the mechanism works, not how good any model is.
A run is a state machine#
Start with the shape. A workflow is a fixed sequence of named steps that turns an input into a result. For example, the ten steps that turn a submission into a recommendation and an underwriter's decision. A step is one named unit of that workflow, such as get_roof_assessment.
After each step the run is in a named state. After load_submission it is received; after get_roof_assessment it is roof_assessed. A state machine is exactly that: a set of named states and the steps that move from one to the next. Writing the run this way is what makes it resumable, because "where were we?" has a short, exact answer: the name of the state and the next step.
| # | Step | Kind | State after it |
|---|---|---|---|
| 1 | load_submission | read | received |
| 2 | get_roof_assessment | outside call (counts; can fail) | roof_assessed |
| 3 | search_guidance | retrieval | guidance_found |
| 4 | find_similar_decisions | retrieval (reads episodic memory) | history_found |
| 5 | check_eligibility | deterministic code | rules_checked |
| 6 | write_recommendation | scripted stand-in for the model | recommended |
| 7 | record_recommendation | write, carries the idempotency key | recorded |
| 8 | notify_cockpit | message through the outbox | notified |
| 9 | underwriter_review | interrupt: the run pauses for a person | waiting_for_underwriter, then decided |
| 10 | remember | memory write policy runs on the finished run | closed |
The third column is the one to keep an eye on. Some steps only read. One step calls the outside world and costs money. Two steps change something outside the run. One step is plain code that always gives the same answer. That column decides, later on this page, what a resume and a replay may do with each step.
For a backend engineer. If you have built an order service with statusesCREATED,PAID,SHIPPED, you have built this. The difference is the length of each transition: an order moves between statuses in milliseconds inside one request, while a run spends seconds or minutes in a step, calling slow services and a model. That is why the run, unlike the order, must save itself between steps.
A checkpoint after every step#
A checkpoint is saved execution state that allows a workflow to resume. In the example it is one row in the checkpoints table, written after each step, holding the state name, the next step and the run state as JSON. This is the schema, from schema.sql:
CREATE TABLE IF NOT EXISTS checkpoints (
checkpoint_id TEXT PRIMARY KEY, -- cp-run-7f3a-0002
run_id TEXT NOT NULL REFERENCES runs(run_id),
thread_id TEXT NOT NULL,
state_version INTEGER NOT NULL, -- 1, 2, 3 ... inside one run
step TEXT NOT NULL, -- the step that just finished (or is waiting)
state_name TEXT NOT NULL, -- roof_assessed, waiting_for_underwriter, ...
next_step TEXT, -- where a resume continues; NULL when the run is closed
attempt_number INTEGER NOT NULL, -- which attempt saved it
state_json TEXT NOT NULL, -- the run state: what the run has gathered so far
created_at TEXT NOT NULL,
UNIQUE (run_id, state_version)
);
The part of the code that saves checkpoints is called a checkpointer. In the example it is the function save_checkpoint in workflow.py. It appends a row with state_version one higher than the last, in one transaction with the update of the run's state name. A checkpoint is appended, never overwritten, so every earlier state of the run stays readable.
Here are the checkpoints of UW-M01, printed by the runner with --verbose. Notice which attempt saved each one.
checkpoints:
cp-run-7f3a-0001 v1 after load_submission → received next get_roof_assessment (attempt 1)
cp-run-7f3a-0002 v2 after get_roof_assessment → roof_assessed next search_guidance (attempt 1)
cp-run-7f3a-0003 v3 after search_guidance → guidance_found next find_similar_decisions (attempt 2)
cp-run-7f3a-0004 v4 after find_similar_decisions → history_found next check_eligibility (attempt 2)
cp-run-7f3a-0005 v5 after check_eligibility → rules_checked next write_recommendation (attempt 2)
cp-run-7f3a-0006 v6 after write_recommendation → recommended next record_recommendation (attempt 2)
cp-run-7f3a-0007 v7 after record_recommendation → recorded next notify_cockpit (attempt 2)
cp-run-7f3a-0008 v8 after notify_cockpit → notified next underwriter_review (attempt 2)
cp-run-7f3a-0009 v9 after underwriter_review → waiting_for_underwriter next underwriter_review (attempt 2)
Attempt 1 saved versions 1 and 2, then died. Attempt 2 loaded version 2 and saved the rest. The state_version counts across attempts, because it belongs to the run, not to a process. The full resume, and why a crash between a step and its checkpoint makes the step run twice, are on Failure and resume.
What this is not. A trace explains a run. A checkpoint resumes it. The two traces of run-7f3a say what each attempt did and how long each step took, for people who diagnose. They are evidence, owned by observe.sammuthu.com. A worker never resumes from a trace: a trace may be sampled, delayed or dropped, and nothing guarantees it holds the state.
The other way: save what happened, not where you are#
A snapshot after each step is the simplest design. There is a second one, and it is worth knowing because it answers questions the snapshot cannot.
An event log is the ordered, append-only record of what happened in a run. For example, "step get_roof_assessment completed, and it added provider_roof_age: 22 and evidence EV-ROOF-0412 to the state". Event sourcing is the design in which that log is the record, and the state is computed by reading the events in order and applying each one. That computation is called folding.
The example keeps both: checkpoints for a fast resume, and an event log for audit and replay. The fold is short. This is fold_events in events.py:
def fold_events(events: list[dict[str, Any]]) -> dict[str, Any]:
"""Rebuild the run state from events, in order. A step that ran again replaces its own patch."""
state: dict[str, Any] = {}
state_name, last_step = None, None
for e in events:
if e["type"] == "step_completed":
state.update(e["payload"]["patch"])
state_name, last_step = e["payload"]["state_name"], e["step"]
return {"state": state, "state_name": state_name, "last_step": last_step}
Folding all the events of run-7f3a gives the same state as checkpoint version 9. That equality is a useful test: if the fold and the newest checkpoint ever disagree, one of them was written wrong.
| Snapshot checkpoints | Event log (event sourcing) | |
|---|---|---|
| What is stored | The whole state after each step | What each step added or changed |
| Resume | Load the newest row | Fold all events, or the events since a snapshot |
| Answers "how did we get here?" | Only by comparing snapshots | Directly, event by event |
| Storage | Grows with state size times steps | Grows with the number of events |
| Risk | Large state copied many times | A change to the fold changes every rebuilt state |
For a backend engineer. This is the same split as a table with the current value versus a Kafka topic of changes with a consumer that builds the current value. Event sourcing in an order service keepsOrderPlaced,PaymentCaptured,OrderShippedand folds them into the order. Teams that use it keep snapshots too, so that a rebuild does not have to start from the first event; the example does the same.
Replay, and the line it must not cross#
To replay is to rebuild a run's state by reading its recorded events again, without redoing their effects. For example, folding the events of run-7f3a up to number 7 shows exactly what the run knew when attempt 1 died.
Replay is safe only if it never does anything. Picture a replay that "re-runs" step 2 to get the roof age: it calls the provider a second time, which costs money, and the provider may now say 23 years instead of 22. Picture a replay that re-runs step 7: a second recommendation row. A replay that re-runs step 6, the model's recommendation, may get a different recommendation, because a model can answer differently to the same input.
So each event gets one of three treatments. Read back: use the recorded result, never redo the step. Recompute: run the step again, because it is deterministic code without side effects, and the result is guaranteed to be the same. Bookkeeping: events about the run itself (attempt started, checkpoint saved) that carry no state. The replay boundary is the line between what may be recomputed and what must be read back.
Determinism is the property that the same input always produces the same output. In the example only check_eligibility is deterministic and has no side effects: the rules are code, and the same facts always give the same result. Everything else is read back. This is replay_action in events.py:
def replay_action(event_type: str, step: str | None) -> str:
"""What replay does with one event: recompute, read_back or bookkeeping."""
if event_type in BOOKKEEPING:
return "bookkeeping"
if event_type == "step_completed" and step in RECOMPUTE_STEPS:
return "recompute"
return "read_back"
Here is the event log of UW-M01, with the replay action of each event, from the runner's --verbose output.
event log (replay action):
1 a- run_started bookkeeping
2 a1 attempt_started bookkeeping
3 a1 step_completed load_submission read_back
4 a1 checkpoint_saved load_submission bookkeeping
5 a1 call_attempt get_roof_assessment read_back
6 a1 step_completed get_roof_assessment read_back
7 a1 checkpoint_saved get_roof_assessment bookkeeping
8 a1 attempt_crashed bookkeeping
9 a2 attempt_started bookkeeping
10 a2 step_completed search_guidance read_back
11 a2 checkpoint_saved search_guidance bookkeeping
12 a2 step_completed find_similar_decisions read_back
13 a2 checkpoint_saved find_similar_decisions bookkeeping
14 a2 step_completed check_eligibility recompute
15 a2 checkpoint_saved check_eligibility bookkeeping
16 a2 step_completed write_recommendation read_back
17 a2 checkpoint_saved write_recommendation bookkeeping
18 a2 recommendation_recorded record_recommendation read_back
19 a2 step_completed record_recommendation read_back
20 a2 checkpoint_saved record_recommendation bookkeeping
21 a2 message_enqueued notify_cockpit read_back
22 a2 step_completed notify_cockpit read_back
23 a2 checkpoint_saved notify_cockpit bookkeeping
24 a2 checkpoint_saved underwriter_review bookkeeping
25 a2 run_paused bookkeeping
Even load_submission is read back, though it only reads. The reason is subtle and worth stating: intake may have corrected the submission since the run read it, so reading it again would give the replay a different input than the run had. A replay answers "what did the run know?", not "what would it know now?".
Step through the same log in the lab. The slider moves through the events; the panel shows the state folded up to that point and, for each event, whether replay reads it back or recomputes it.
| Events | Step | Replay action | Redoing it instead would cause |
|---|---|---|---|
| 3 | load_submission | read back | a submission intake may have corrected since |
| 5, 6 | get_roof_assessment | read back | a second call to the outside provider |
| 10 | search_guidance | read back | a different answer if the guidance index changed |
| 12 | find_similar_decisions | read back | a different answer if memory changed since |
| 14 | check_eligibility | recompute | nothing: deterministic code gives the same result |
| 16 | write_recommendation | read back | a different recommendation: a model may answer differently |
| 18, 19 | record_recommendation | read back | a second recommendation row without a key |
| 21, 22 | notify_cockpit | read back | a second message to the cockpit |
The diagram draws the boundary through the ten steps.
Diagram: The replay boundary through the ten steps · read back or recompute
Read left to right. Steps that touched the outside world, read a store that can change, or asked a model are read back from the record and never redone. Only deterministic code without side effects may be recomputed. The pause for a person is neither: it waits for a decision that only a person writes.
- Read back: touched the world
- get_roof_assessment: Redoing it is a second paid call, and the answer may differ.
- record_recommendation: Redoing it is a second row without a key.
- notify_cockpit: Redoing it is a second message to the cockpit.
- Read back: may have changed
- load_submission: Intake may have corrected the submission since; replay must see what the run saw.
- search_guidance, find_similar_decisions: The index and memory can change; the recorded answer is what the run used.
- write_recommendation: A model may answer differently to the same input.
- Recompute
- check_eligibility: The rules are code: the same facts always give the same result, and nothing outside changes.
- Waits for a person
- underwriter_review: The run saves its state and stops until the underwriter decides.
Read it from left to right. Most of the run sits on the "read back" side: anything that touched the world, read a store that can change, or asked a model. Only the deterministic rules sit on the "recompute" side. That is the normal shape of an agent workflow, and it is the opposite of a pure data pipeline, where most steps can be recomputed.
The interrupt: a pause that lasts three days#
Step 9 is different from the others. The run cannot continue until a person decides. An interrupt is a planned pause in which the run saves its state and stops, waiting for an outside input, here the underwriter's decision.
Case UW-M06: run run-5f17 for SUB-1009 recommends approve and reaches underwriter_review. No decision exists yet, so the step returns a pause, the checkpoint is saved with state waiting_for_underwriter, and the attempt ends with status paused. No process waits for three days. Three days later the underwriter declines and records a reason; the run resumes from the checkpoint as attempt 2, reads the decision, and goes on to remember.
UW-M06 The run waits three days for the underwriter, who declines against an approve recommendation
run run-5f17 · thread-SUB-1009 · corr-7b36 · SUB-1009 (BRK-017)
attempt 1 trace 3305bb48… paused from the start steps run: load_submission, get_roof_assessment, search_guidance, find_similar_decisions, check_eligibility, write_recommendation, record_recommendation, notify_cockpit
attempt 2 trace efee4a1a… ok from state_version 9 steps run: underwriter_review, remember
provider calls 1 · recommendation rows 1 · approve citing no rule · run completed
expected (built on UW-009): PASS · 12 of 12 checks
Attempt 2 ran only two steps. Nothing before the interrupt ran again, and the provider was called once. The decision row is written by the person, through the cockpit, never by the agent. The model recommends; a person decides.
Common confusion. A pause is not a crash, though both end an attempt and both are resumed from a checkpoint. A crash is unplanned and its last step may be half done. A pause is planned and happens at a step boundary, so nothing is half done.
The failure, shown#
Without checkpoints, a crash after step 2 sends the run back to step 1 and the provider is called twice. With checkpoints but a replay that recomputes everything, a debugging session that replays run-7f3a to see what went wrong calls the provider, writes a second recommendation and sends a second message: the investigation changes the thing it investigates. With checkpoints and the boundary drawn, a resume skips finished steps and a replay is read-only.
Framework implementation#
Some frameworks save a checkpoint after every step for you and let you resume a thread; some durable workflow engines are built on an event history that they replay, and require workflow code to be deterministic for that reason. Which named framework does which, and what it requires of your code, is on Frameworks. The replay boundary is the question to ask of each one: what does it re-execute, and what does it read back?
A checklist for Monday#
- Can you name every state of your run, and the next step from each?
- Where is the checkpoint written, and is it in the same transaction as the step's own writes?
- Is each checkpoint appended with a version, or overwritten in place?
- For each step, is it read back or recomputed on replay? Is the answer written down in code, as in
replay_action? - Does folding your event log give the same state as your newest checkpoint? Test it.
- When the run waits for a person, does any process wait with it?
Advanced: how often to checkpoint, and how much to save
A checkpoint after every step is the default because it bounds the work lost in a crash to one step. It costs one write per step and stores the whole state each time. For the example's ten steps and small state, that cost is negligible. For a run with hundreds of model calls and a large working set, writing the whole state each time becomes expensive, and two designs help: checkpoint only after steps with side effects or high cost (and recompute or re-read the cheap ones), or store deltas, which is event sourcing with periodic snapshots.
What goes into the state matters as much as when. Store references, not copies: the evidence id EV-ROOF-0412 and not the provider's full report, a retrieved passage's id and not its text. The state then stays small, and the large objects live once, in stores built for them. The one thing that must be in the checkpoint is whatever a resumed step needs and cannot get again identically, which includes the model's output from step 6.
Advanced: determinism is a property you must protect
check_eligibility is deterministic today. It stops being deterministic the day someone adds a call to the current date, a lookup in a table that changes, or a random tie-break. A replay then silently computes a different eligibility result than the run had. Two habits protect the boundary. Pass time, versions and reference data into deterministic steps as inputs, recorded in the state, instead of reading them inside the step. And test the property: replay a stored run and assert that every recomputed step reproduces its recorded result exactly. Engines that replay workflow code from history enforce a stricter version of this rule on all workflow code; that is why they ask for activities, the steps with side effects, to be separated from it.
In one breath#
A run is a state machine: named steps, a named state after each. A checkpoint saves that state after every step so that a run can resume; an event log saves what each step did so that the state can be rebuilt and explained. Replay rebuilds without acting: everything that touched the world, read something changeable or asked a model is read back, and only deterministic code may be recomputed. A trace explains a run. A checkpoint resumes it.
The same idea for a coding agent: the journal file says where the task stopped, the commit history is its event log, and replaying the log must never re-run the migration it records.
The record now says where the run stopped and what each step did. The next page uses it to do the hard part: continue a crashed run without repeating its side effects. Failure and resume.