This page deepens the whole site. It assumes the vocabulary of the earlier pages: a run, an attempt, a checkpoint, an idempotency key, the outbox, a memory with provenance and expiry. If a word is new, the glossary link on it leads to its home.
Run run-9d20 for SUB-1009 has been paused for three days. It is case UW-M06, one of this site's own cases: the agent recommended approve, and the run waits at the step underwriter_review for a person to decide. Its last checkpoint holds state_version 8 and a state object written by version 1 of the workflow code.
On the second day the team ships version 2. It adds a step check_flood_zone between check_eligibility and write_recommendation, and it renames the state field roof_assessment to roof. On the third day the underwriter declines, and the worker running version 2 loads the checkpoint to continue.
The new code looks for roof and finds nothing. It sees a run that skipped a step which, for this run, never existed. Nothing in the earlier pages prepared for this. Every mechanism there assumed that the code that wrote the state is the code that reads it. In production that assumption fails every time you deploy while runs are in flight. The rest of this page deals with five problems of that kind, where the system outlives one version of itself.
The agent here is a scripted stand-in, so the results on this page show how the mechanism works, not how good any model is.
1. Saved state meets new code#
A checkpoint is saved execution state that allows a workflow to resume. It is a contract between the code that wrote it and the code that will read it. Versioned state means the checkpoint carries the version of the state's shape, for example "state_schema": 1, so that a reader knows which contract it holds.
State migration is changing saved state from an old shape to a new one. For example, renaming roof_assessment to roof in every checkpoint of every paused run, or doing it as each one is loaded.
There are three designs, and each has a cost.
| Design | How it works | Cost | Good when |
|---|---|---|---|
| Drain | Stop starting runs on v1; let them finish; then switch | Long-paused runs block the release, sometimes for days | Runs are short |
| Migrate on read | v2 reads state_schema, upgrades a v1 state in memory (an upcaster), continues | Every shape change needs an upcaster and a test; they accumulate | Shapes change often, runs are long |
| Run both | v1 workers finish v1 runs; v2 workers take new runs | Two code versions in production until the last v1 run ends | Changes to the step sequence, not only the shape |
The rename is a shape change; an upcaster fixes it. The new step is a sequence change, and it is harder. For run-9d20, step 6 already ran under v1 without the flood check. Should v2 run check_flood_zone now, after the recommendation was written? The answer depends on the business, not the code: the recommendation was made without a flood check, and it is the underwriter who decides. A safe default is to finish old runs on the old sequence and record the code version on every checkpoint, so that an auditor can see which sequence a run followed.
Diagram: A paused run meets new code · read the version before reading the state
Read left to right. The checkpoint was written under version 1 and says so. The new worker reads the schema number first. A shape change is upgraded in memory by an upcaster and the run continues under version 2. A change to the sequence of steps sends the run to a version 1 worker, which finishes the old sequence. No checkpoint is read by code that does not understand it.
- 1 · Saved under v1
- Checkpoint: The paused run's saved state, carrying the version of its shape and of the code that wrote it.
- 2 · Read the version
- Version check: The worker under version 2 reads the schema number before any field.
- 3a · Shape changed
- Upcaster: A small function that turns an old shape into the new one when the state is loaded.
- 3b · Steps changed
- v1 worker: Runs started under v1 finish on the v1 sequence; the code version is recorded for audit.
Read the diagram left to right. A checkpoint written under v1 carries its schema number. The reader checks it. A shape change goes through an upcaster and continues under v2. A sequence change routes the run to a v1 worker that finishes the old sequence. In both paths the checkpoint is never read by code that does not understand it.
For a backend engineer. This is a database schema migration with a twist. A table migration runs once, in a transaction. Saved run state is often a JSON blob in many rows, each read at an unknown future time. It behaves more like a serialized Java object in a long-lived session: change the class, and old serialized instances must still deserialize. The habit that helps is the same one serialization teaches: a version field in the payload and a reader that handles every version still in storage.
Durable workflow engines that replay a run's history to rebuild state are the strictest case: a code change that alters the order of recorded steps can make an old history unreadable by the new code. How a named engine handles this is described from its own documentation on Frameworks.
2. Two memories that disagree#
The memory store of the example holds MEM-0031 in scope broker:BRK-017: "This broker reports roof age from the owner's statement, not from an inspection", written from an underwriter's decision, assigned confidence 0.95. A month later the broker writes: "From now on our roof ages come from inspection reports." The write policy stores that as a broker statement, assigned confidence 0.5. Both are active. They contradict each other.
Conflict resolution is the rule that decides what a reader is told when two memories disagree. There are four common rules.
| Rule | What the reader gets | Risk |
|---|---|---|
| Latest wins | The broker statement | A low-trust source replaces a high-trust one by being newer |
| Most trusted wins | The underwriter's memory | A true change is ignored until someone with authority confirms it |
| Both, flagged | Both, marked as a conflict | The reader, or the underwriter, must resolve it |
| Supersede on write | The policy marks the old one superseded when the new one is stored | Only safe when the new source is at least as trusted |
For memories that change a recommendation, "both, flagged" is the honest default: the agent cites neither as a reason and lists the conflict for the underwriter, who is the person with authority to settle it. Her decision then becomes a new memory from source underwriter_decision, which may supersede the old one. "Supersede on write" is right only in the narrow case where the new source is at least as trusted as the old: a later underwriter decision may supersede an earlier one; a broker statement may not supersede an underwriter decision.
The write policy enforces this with a uniqueness rule: at most one active memory per scope and subject. A second candidate for broker:BRK-017 and roof_age_source cannot be silently added. It must either supersede the first, if its source is trusted enough, or be stored as a conflict for review.
3. Forgetting as a design#
Forgetting means removing or downgrading memories on purpose. The earlier pages have three forms of it already: expiry (a broker statement stops being returned after 30 days), deletion with a tombstone (on request or by retention), and the recency signal in ranking (an old memory scores lower). They are different tools.
| Form | What it does | When it is right |
|---|---|---|
| Expiry | The memory is no longer returned after a date | The information goes stale on a known schedule |
| Decay in ranking | The memory is still returned, but ranks lower with age | Older information is less likely, not less true |
| Supersession | A newer memory replaces it; the old one stays for audit | The information changed |
| Deletion | The text is removed; a tombstone remains | The information must not exist, by request or schedule |
The design question is which form each kind of memory gets, and it is decided per kind, not per store. An underwriter decision that set a semantic memory should decay slowly and be superseded explicitly. A broker statement should expire. A preference should live until the person changes it. Episodic memory is rarely forgotten, but it must be read as history once its policy version is old.
A store that never forgets does not become wiser. It becomes slower to search and more likely to return something stale ahead of something current. A store that forgets by accident, when a cache is cleared or a window rolls over, is worse: it loses what was supposed to stay. Forgetting deserves the same deliberate design as writing.
4. Summaries of summaries#
A summary is a lossy copy. History or facts showed what one summary loses. Summary drift is what happens when the summary is itself summarized again, together with new material, round after round.
Take the twelve-message exchange about SUB-1003. Round 1 summarizes turns 1 to 8: "The broker states the roof was replaced in 2019; invoice to follow." Round 2 summarizes that summary plus turns 9 to 12: "Roof replaced 2019; invoice pending." Round 3, a week later: "Roof replaced 2019." In three rounds a claim with a pending proof became a plain fact, with no single step that anyone would call wrong.
Three habits keep drift bounded. Summarize from the original turns, not from the previous summary, whenever the originals are still kept. Keep claims as durable facts with a source type, outside the summary, so that their status cannot be compressed away. And link each summary to the range of turns it covers, so that any statement in it can be checked against its source.
5. Losing the database: backup, restore and disaster recovery#
Disaster recovery is bringing a system back after losing a whole store or site, not one process. For example, the database of the example is lost along with its disk, and the last backup is from 03:00.
Two numbers frame the plan. The recovery point is how far back the restored data is: a recovery point objective (RPO) is the maximum acceptable time between the last recovery point and the failure, which is the data you accept losing. The recovery time objective (RTO) is the maximum acceptable delay before service is back. Both are set by the organization, not by the technology. A backup at 03:00 and a failure at 10:00 means up to seven hours of runs, checkpoints and memories are gone.
The tables of this site do not fail independently, and that is the hard part. Consider what a restore to 03:00 does to run run-7f3a, which finished at 09:40.
| Table | After the restore | Consequence |
|---|---|---|
runs, checkpoints | run-7f3a does not exist | The submission looks untouched; a new run starts |
recommendations | No row | The new run writes a recommendation again |
outbox | No row | Fine: no message pending |
| Cockpit inbox (another system, not restored) | Holds the 09:41 message for run-7f3a | The underwriter already saw a recommendation the insurer's store no longer has |
memories | Memories written after 03:00 are gone; tombstones written after 03:00 are gone | A note deleted at 08:00 on request is back |
The last row is the dangerous one. An erasure that was carried out is undone by a restore, silently. The repair is to keep the deletion log outside the main backup cycle, or with a recovery point shorter than the main one, and to replay it after every restore. The same holds for the cockpit's view: the restore plan must say what the systems that were not restored will see, and how the gap is reconciled. Here the reconciliation is to compare the cockpit's inbox with the restored recommendations table and re-import what the cockpit holds, with its original idempotency keys, so that the new run does not write a second recommendation.
Production pitfall. A backup that has never been restored is a hope, not a recovery plan. Restore into a separate environment on a schedule, run the recovery test from Evaluation against it, and check that deleted memories are still deleted.
How the five fit together#
All five problems share one cause: the system lives longer than any single version of its code, its data or its people's intentions. Saved state outlives the code that wrote it. A memory outlives the situation it described. A summary outlives the turns it compressed. A backup outlives the deletions made after it. Each fix makes the version, the source or the time explicit, so that a reader can tell what it holds.
A checklist for a Principal engineer#
- Does every checkpoint carry the schema version of its state and the code version that wrote it?
- What happens to a paused run during a deploy that changes the step sequence? Is that tested?
- Is there at most one active memory per scope and subject, and what happens to the second candidate?
- For each kind of memory, which form of forgetting applies?
- Are summaries built from originals, and is each linked to the range it covers?
- What are the RPO and RTO, who set them, and when was the last restore test?
- After a restore, how are deletions re-applied, and how are systems that were not restored reconciled?
Deeper: testing a state migration
The test that matters loads real old states, not freshly made ones. Keep a corpus of checkpoints from every state schema still present in storage, made by running the old code on the cases of the example, and run every upcaster over every one in CI. Then resume each upgraded state under the new code and compare the end state with an uninterrupted run under the new code, the end-state equivalence of Evaluation. An upcaster may be removed only when no stored state of that schema remains, which a query over the checkpoint store can prove.
Deeper: restoring tables to one consistent point
The tables that implement the earlier pages, checkpoints, recommendations, outbox, the idempotency records and the consumer's inbox, are correct only together. If they live in one database and are backed up as one consistent snapshot, a restore keeps them consistent with each other, and the problem moves to systems outside that database. If they live in different stores with different backup times, a restore can produce a recommendation without its outbox row, or an outbox row whose message the consumer's restored inbox no longer remembers, which will then be processed twice. That is a strong argument for keeping the run's state, its effects' records and the outbox in one transactional store, the design this site uses, and treating everything outside it as a consumer that deduplicates.
In one breath#
Production systems outlive their own versions. Saved state needs a schema version and a migration path, and a change to the step sequence is harder than a change to the shape. Two memories that disagree need a rule, usually "both, flagged" for a person, and a uniqueness rule that turns contradictions into visible conflicts. Forgetting is chosen per kind. Summaries are rebuilt from originals. And a restore is part of the design: it must keep the run's tables consistent and re-apply deletions.
The same idea for a coding agent: a long session resumed after the repository's instruction file changed is a migration; two instruction files that contradict each other need a precedence rule; and an old compacted summary of the session is the summary drift of this page.
This is the end of the series of chapters. The Insurance lab shows the whole example in one place, and the Glossary collects every term.