All chapters
  1. Start hereMemory is what you keep. A checkpoint is where you stopped
  2. Memory · 1The stateless request: why anything has to be stored at all
  3. Memory · 2Five kinds of memory: working, episodic, semantic, procedural and preference
  4. Memory · 3Conversation history, summaries, retrieval and durable facts
  5. Memory · 4Files as memory in coding agents: instructions, journals, artifacts and checkpoints
  6. Memory · 5Database stores: tables, JSON, documents, objects and vectors
  7. State · 6Run identity: run_id, thread_id, trace_id, correlation_id and task_id
  8. State · 7State machines, checkpoints, event sourcing and where replay must stop
  9. State · 8Failure and resume: idempotency keys, retry policy and the exactly-once illusion
  10. State · 9The transactional outbox: side effects, compensation and deduplication
  11. Policy · 10Memory write policy: extraction, confidence, provenance, expiry and deletion
  12. Policy · 11Retrieval and ranking: relevance, recency, salience and scope
  13. Policy · 12Shared memory between agents: consistency, isolation, ownership and races
  14. Policy · 13Security: tenant separation, personal data, secrets, consent and retention
  15. Practice · 14Evaluation: memory correctness, stale facts, contamination and recovery tests
  16. Practice · 15Frameworks and standards: what each one stores for you
  17. Practice · 16The insurance lab: one run that fails, resumes and remembers, end to end
  18. Practice · 17Advanced: migration, conflict resolution, forgetting, summary drift and disaster recovery
  19. ReferenceGlossary
  20. AboutAbout this site and its sources
State · 7

State machines, checkpoints, event sourcing and where replay must stop

A run is a sequence of named states. Saving the state after each step, or saving the events that produced it, and the steps that must never be replayed.

What you will understand
A run as states
Ten named steps, one named state after each: a state machine you can save and reload.
Snapshot or events
Save the state after each step, or save what happened and fold it back into state.
Where replay stops
Which steps may be recomputed, and which must be read from the record and never redone.

The worker running run-7f3a for SUB-1004 has finished two steps. It loaded the submission, and it called the outside provider, which said the roof is 22 years old. At that moment the process dies. The previous page gave the run a name that outlives the process. A name is not enough, though. The next worker needs to know three things: which step comes next, what the run has gathered so far, and which of the finished steps did something outside the run that must not happen twice.

This page builds the record that answers those three questions. It starts with the shape of the run itself, then compares two ways to save it, then draws the line that matters most: what may be computed again, and what must be read back. The case is UW-M01, the anchor of this site's own cases; the pause for a person is UW-M06.

The agent here is a scripted stand-in, so the results on this page show how the mechanism works, not how good any model is.

A run is a state machine#

Start with the shape. A workflow is a fixed sequence of named steps that turns an input into a result. For example, the ten steps that turn a submission into a recommendation and an underwriter's decision. A step is one named unit of that workflow, such as get_roof_assessment.

After each step the run is in a named state. After load_submission it is received; after get_roof_assessment it is roof_assessed. A state machine is exactly that: a set of named states and the steps that move from one to the next. Writing the run this way is what makes it resumable, because "where were we?" has a short, exact answer: the name of the state and the next step.

#StepKindState after it
1load_submissionreadreceived
2get_roof_assessmentoutside call (counts; can fail)roof_assessed
3search_guidanceretrievalguidance_found
4find_similar_decisionsretrieval (reads episodic memory)history_found
5check_eligibilitydeterministic coderules_checked
6write_recommendationscripted stand-in for the modelrecommended
7record_recommendationwrite, carries the idempotency keyrecorded
8notify_cockpitmessage through the outboxnotified
9underwriter_reviewinterrupt: the run pauses for a personwaiting_for_underwriter, then decided
10remembermemory write policy runs on the finished runclosed

The third column is the one to keep an eye on. Some steps only read. One step calls the outside world and costs money. Two steps change something outside the run. One step is plain code that always gives the same answer. That column decides, later on this page, what a resume and a replay may do with each step.

For a backend engineer. If you have built an order service with statuses CREATED, PAID, SHIPPED, you have built this. The difference is the length of each transition: an order moves between statuses in milliseconds inside one request, while a run spends seconds or minutes in a step, calling slow services and a model. That is why the run, unlike the order, must save itself between steps.

A checkpoint after every step#

A checkpoint is saved execution state that allows a workflow to resume. In the example it is one row in the checkpoints table, written after each step, holding the state name, the next step and the run state as JSON. This is the schema, from schema.sql:

CREATE TABLE IF NOT EXISTS checkpoints (
  checkpoint_id   TEXT PRIMARY KEY,                 -- cp-run-7f3a-0002
  run_id          TEXT NOT NULL REFERENCES runs(run_id),
  thread_id       TEXT NOT NULL,
  state_version   INTEGER NOT NULL,                 -- 1, 2, 3 ... inside one run
  step            TEXT NOT NULL,                    -- the step that just finished (or is waiting)
  state_name      TEXT NOT NULL,                    -- roof_assessed, waiting_for_underwriter, ...
  next_step       TEXT,                             -- where a resume continues; NULL when the run is closed
  attempt_number  INTEGER NOT NULL,                 -- which attempt saved it
  state_json      TEXT NOT NULL,                    -- the run state: what the run has gathered so far
  created_at      TEXT NOT NULL,
  UNIQUE (run_id, state_version)
);

The part of the code that saves checkpoints is called a checkpointer. In the example it is the function save_checkpoint in workflow.py. It appends a row with state_version one higher than the last, in one transaction with the update of the run's state name. A checkpoint is appended, never overwritten, so every earlier state of the run stays readable.

Here are the checkpoints of UW-M01, printed by the runner with --verbose. Notice which attempt saved each one.

  checkpoints:
    cp-run-7f3a-0001  v1  after load_submission        → received                 next get_roof_assessment    (attempt 1)
    cp-run-7f3a-0002  v2  after get_roof_assessment    → roof_assessed            next search_guidance        (attempt 1)
    cp-run-7f3a-0003  v3  after search_guidance        → guidance_found           next find_similar_decisions (attempt 2)
    cp-run-7f3a-0004  v4  after find_similar_decisions → history_found            next check_eligibility      (attempt 2)
    cp-run-7f3a-0005  v5  after check_eligibility      → rules_checked            next write_recommendation   (attempt 2)
    cp-run-7f3a-0006  v6  after write_recommendation   → recommended              next record_recommendation  (attempt 2)
    cp-run-7f3a-0007  v7  after record_recommendation  → recorded                 next notify_cockpit         (attempt 2)
    cp-run-7f3a-0008  v8  after notify_cockpit         → notified                 next underwriter_review     (attempt 2)
    cp-run-7f3a-0009  v9  after underwriter_review     → waiting_for_underwriter  next underwriter_review     (attempt 2)

Attempt 1 saved versions 1 and 2, then died. Attempt 2 loaded version 2 and saved the rest. The state_version counts across attempts, because it belongs to the run, not to a process. The full resume, and why a crash between a step and its checkpoint makes the step run twice, are on Failure and resume.

What this is not. A trace explains a run. A checkpoint resumes it. The two traces of run-7f3a say what each attempt did and how long each step took, for people who diagnose. They are evidence, owned by observe.sammuthu.com. A worker never resumes from a trace: a trace may be sampled, delayed or dropped, and nothing guarantees it holds the state.

The other way: save what happened, not where you are#

A snapshot after each step is the simplest design. There is a second one, and it is worth knowing because it answers questions the snapshot cannot.

An event log is the ordered, append-only record of what happened in a run. For example, "step get_roof_assessment completed, and it added provider_roof_age: 22 and evidence EV-ROOF-0412 to the state". Event sourcing is the design in which that log is the record, and the state is computed by reading the events in order and applying each one. That computation is called folding.

The example keeps both: checkpoints for a fast resume, and an event log for audit and replay. The fold is short. This is fold_events in events.py:

def fold_events(events: list[dict[str, Any]]) -> dict[str, Any]:
    """Rebuild the run state from events, in order. A step that ran again replaces its own patch."""
    state: dict[str, Any] = {}
    state_name, last_step = None, None
    for e in events:
        if e["type"] == "step_completed":
            state.update(e["payload"]["patch"])
            state_name, last_step = e["payload"]["state_name"], e["step"]
    return {"state": state, "state_name": state_name, "last_step": last_step}

Folding all the events of run-7f3a gives the same state as checkpoint version 9. That equality is a useful test: if the fold and the newest checkpoint ever disagree, one of them was written wrong.

Snapshot checkpointsEvent log (event sourcing)
What is storedThe whole state after each stepWhat each step added or changed
ResumeLoad the newest rowFold all events, or the events since a snapshot
Answers "how did we get here?"Only by comparing snapshotsDirectly, event by event
StorageGrows with state size times stepsGrows with the number of events
RiskLarge state copied many timesA change to the fold changes every rebuilt state
For a backend engineer. This is the same split as a table with the current value versus a Kafka topic of changes with a consumer that builds the current value. Event sourcing in an order service keeps OrderPlaced, PaymentCaptured, OrderShipped and folds them into the order. Teams that use it keep snapshots too, so that a rebuild does not have to start from the first event; the example does the same.

Replay, and the line it must not cross#

To replay is to rebuild a run's state by reading its recorded events again, without redoing their effects. For example, folding the events of run-7f3a up to number 7 shows exactly what the run knew when attempt 1 died.

Replay is safe only if it never does anything. Picture a replay that "re-runs" step 2 to get the roof age: it calls the provider a second time, which costs money, and the provider may now say 23 years instead of 22. Picture a replay that re-runs step 7: a second recommendation row. A replay that re-runs step 6, the model's recommendation, may get a different recommendation, because a model can answer differently to the same input.

So each event gets one of three treatments. Read back: use the recorded result, never redo the step. Recompute: run the step again, because it is deterministic code without side effects, and the result is guaranteed to be the same. Bookkeeping: events about the run itself (attempt started, checkpoint saved) that carry no state. The replay boundary is the line between what may be recomputed and what must be read back.

Determinism is the property that the same input always produces the same output. In the example only check_eligibility is deterministic and has no side effects: the rules are code, and the same facts always give the same result. Everything else is read back. This is replay_action in events.py:

def replay_action(event_type: str, step: str | None) -> str:
    """What replay does with one event: recompute, read_back or bookkeeping."""
    if event_type in BOOKKEEPING:
        return "bookkeeping"
    if event_type == "step_completed" and step in RECOMPUTE_STEPS:
        return "recompute"
    return "read_back"

Here is the event log of UW-M01, with the replay action of each event, from the runner's --verbose output.

  event log (replay action):
      1  a-  run_started                                      bookkeeping
      2  a1  attempt_started                                  bookkeeping
      3  a1  step_completed           load_submission         read_back
      4  a1  checkpoint_saved         load_submission         bookkeeping
      5  a1  call_attempt             get_roof_assessment     read_back
      6  a1  step_completed           get_roof_assessment     read_back
      7  a1  checkpoint_saved         get_roof_assessment     bookkeeping
      8  a1  attempt_crashed                                  bookkeeping
      9  a2  attempt_started                                  bookkeeping
     10  a2  step_completed           search_guidance         read_back
     11  a2  checkpoint_saved         search_guidance         bookkeeping
     12  a2  step_completed           find_similar_decisions  read_back
     13  a2  checkpoint_saved         find_similar_decisions  bookkeeping
     14  a2  step_completed           check_eligibility       recompute
     15  a2  checkpoint_saved         check_eligibility       bookkeeping
     16  a2  step_completed           write_recommendation    read_back
     17  a2  checkpoint_saved         write_recommendation    bookkeeping
     18  a2  recommendation_recorded  record_recommendation   read_back
     19  a2  step_completed           record_recommendation   read_back
     20  a2  checkpoint_saved         record_recommendation   bookkeeping
     21  a2  message_enqueued         notify_cockpit          read_back
     22  a2  step_completed           notify_cockpit          read_back
     23  a2  checkpoint_saved         notify_cockpit          bookkeeping
     24  a2  checkpoint_saved         underwriter_review      bookkeeping
     25  a2  run_paused                                       bookkeeping

Even load_submission is read back, though it only reads. The reason is subtle and worth stating: intake may have corrected the submission since the run read it, so reading it again would give the replay a different input than the run had. A replay answers "what did the run know?", not "what would it know now?".

Step through the same log in the lab. The slider moves through the events; the panel shows the state folded up to that point and, for each event, whether replay reads it back or recomputes it.

EventsStepReplay actionRedoing it instead would cause
3load_submissionread backa submission intake may have corrected since
5, 6get_roof_assessmentread backa second call to the outside provider
10search_guidanceread backa different answer if the guidance index changed
12find_similar_decisionsread backa different answer if memory changed since
14check_eligibilityrecomputenothing: deterministic code gives the same result
16write_recommendationread backa different recommendation: a model may answer differently
18, 19record_recommendationread backa second recommendation row without a key
21, 22notify_cockpitread backa second message to the cockpit

The diagram draws the boundary through the ten steps.

Diagram: The replay boundary through the ten steps · read back or recompute

Read left to right. Steps that touched the outside world, read a store that can change, or asked a model are read back from the record and never redone. Only deterministic code without side effects may be recomputed. The pause for a person is neither: it waits for a decision that only a person writes.

  1. Read back: touched the world
    1. get_roof_assessment: Redoing it is a second paid call, and the answer may differ.
    2. record_recommendation: Redoing it is a second row without a key.
    3. notify_cockpit: Redoing it is a second message to the cockpit.
  2. Read back: may have changed
    1. load_submission: Intake may have corrected the submission since; replay must see what the run saw.
    2. search_guidance, find_similar_decisions: The index and memory can change; the recorded answer is what the run used.
    3. write_recommendation: A model may answer differently to the same input.
  3. Recompute
    1. check_eligibility: The rules are code: the same facts always give the same result, and nothing outside changes.
  4. Waits for a person
    1. underwriter_review: The run saves its state and stops until the underwriter decides.

Read it from left to right. Most of the run sits on the "read back" side: anything that touched the world, read a store that can change, or asked a model. Only the deterministic rules sit on the "recompute" side. That is the normal shape of an agent workflow, and it is the opposite of a pure data pipeline, where most steps can be recomputed.

The interrupt: a pause that lasts three days#

Step 9 is different from the others. The run cannot continue until a person decides. An interrupt is a planned pause in which the run saves its state and stops, waiting for an outside input, here the underwriter's decision.

Case UW-M06: run run-5f17 for SUB-1009 recommends approve and reaches underwriter_review. No decision exists yet, so the step returns a pause, the checkpoint is saved with state waiting_for_underwriter, and the attempt ends with status paused. No process waits for three days. Three days later the underwriter declines and records a reason; the run resumes from the checkpoint as attempt 2, reads the decision, and goes on to remember.

UW-M06  The run waits three days for the underwriter, who declines against an approve recommendation
  run run-5f17 · thread-SUB-1009 · corr-7b36 · SUB-1009 (BRK-017)
  attempt 1  trace 3305bb48…  paused   from the start         steps run: load_submission, get_roof_assessment, search_guidance, find_similar_decisions, check_eligibility, write_recommendation, record_recommendation, notify_cockpit
  attempt 2  trace efee4a1a…  ok       from state_version 9   steps run: underwriter_review, remember
  provider calls 1 · recommendation rows 1 · approve citing no rule · run completed
  expected (built on UW-009): PASS · 12 of 12 checks

Attempt 2 ran only two steps. Nothing before the interrupt ran again, and the provider was called once. The decision row is written by the person, through the cockpit, never by the agent. The model recommends; a person decides.

Common confusion. A pause is not a crash, though both end an attempt and both are resumed from a checkpoint. A crash is unplanned and its last step may be half done. A pause is planned and happens at a step boundary, so nothing is half done.

The failure, shown#

Without checkpoints, a crash after step 2 sends the run back to step 1 and the provider is called twice. With checkpoints but a replay that recomputes everything, a debugging session that replays run-7f3a to see what went wrong calls the provider, writes a second recommendation and sends a second message: the investigation changes the thing it investigates. With checkpoints and the boundary drawn, a resume skips finished steps and a replay is read-only.

Framework implementation#

Some frameworks save a checkpoint after every step for you and let you resume a thread; some durable workflow engines are built on an event history that they replay, and require workflow code to be deterministic for that reason. Which named framework does which, and what it requires of your code, is on Frameworks. The replay boundary is the question to ask of each one: what does it re-execute, and what does it read back?

A checklist for Monday#

  1. Can you name every state of your run, and the next step from each?
  2. Where is the checkpoint written, and is it in the same transaction as the step's own writes?
  3. Is each checkpoint appended with a version, or overwritten in place?
  4. For each step, is it read back or recomputed on replay? Is the answer written down in code, as in replay_action?
  5. Does folding your event log give the same state as your newest checkpoint? Test it.
  6. When the run waits for a person, does any process wait with it?
Advanced: how often to checkpoint, and how much to save

A checkpoint after every step is the default because it bounds the work lost in a crash to one step. It costs one write per step and stores the whole state each time. For the example's ten steps and small state, that cost is negligible. For a run with hundreds of model calls and a large working set, writing the whole state each time becomes expensive, and two designs help: checkpoint only after steps with side effects or high cost (and recompute or re-read the cheap ones), or store deltas, which is event sourcing with periodic snapshots.

What goes into the state matters as much as when. Store references, not copies: the evidence id EV-ROOF-0412 and not the provider's full report, a retrieved passage's id and not its text. The state then stays small, and the large objects live once, in stores built for them. The one thing that must be in the checkpoint is whatever a resumed step needs and cannot get again identically, which includes the model's output from step 6.

Advanced: determinism is a property you must protect

check_eligibility is deterministic today. It stops being deterministic the day someone adds a call to the current date, a lookup in a table that changes, or a random tie-break. A replay then silently computes a different eligibility result than the run had. Two habits protect the boundary. Pass time, versions and reference data into deterministic steps as inputs, recorded in the state, instead of reading them inside the step. And test the property: replay a stored run and assert that every recomputed step reproduces its recorded result exactly. Engines that replay workflow code from history enforce a stricter version of this rule on all workflow code; that is why they ask for activities, the steps with side effects, to be separated from it.

In one breath#

A run is a state machine: named steps, a named state after each. A checkpoint saves that state after every step so that a run can resume; an event log saves what each step did so that the state can be rebuilt and explained. Replay rebuilds without acting: everything that touched the world, read something changeable or asked a model is read back, and only deterministic code may be recomputed. A trace explains a run. A checkpoint resumes it.

The same idea for a coding agent: the journal file says where the task stopped, the commit history is its event log, and replaying the log must never re-run the migration it records.

The record now says where the run stopped and what each step did. The next page uses it to do the hard part: continue a crashed run without repeating its side effects. Failure and resume.