All chapters
  1. Start hereMemory is what you keep. A checkpoint is where you stopped
  2. Memory · 1The stateless request: why anything has to be stored at all
  3. Memory · 2Five kinds of memory: working, episodic, semantic, procedural and preference
  4. Memory · 3Conversation history, summaries, retrieval and durable facts
  5. Memory · 4Files as memory in coding agents: instructions, journals, artifacts and checkpoints
  6. Memory · 5Database stores: tables, JSON, documents, objects and vectors
  7. State · 6Run identity: run_id, thread_id, trace_id, correlation_id and task_id
  8. State · 7State machines, checkpoints, event sourcing and where replay must stop
  9. State · 8Failure and resume: idempotency keys, retry policy and the exactly-once illusion
  10. State · 9The transactional outbox: side effects, compensation and deduplication
  11. Policy · 10Memory write policy: extraction, confidence, provenance, expiry and deletion
  12. Policy · 11Retrieval and ranking: relevance, recency, salience and scope
  13. Policy · 12Shared memory between agents: consistency, isolation, ownership and races
  14. Policy · 13Security: tenant separation, personal data, secrets, consent and retention
  15. Practice · 14Evaluation: memory correctness, stale facts, contamination and recovery tests
  16. Practice · 15Frameworks and standards: what each one stores for you
  17. Practice · 16The insurance lab: one run that fails, resumes and remembers, end to end
  18. Practice · 17Advanced: migration, conflict resolution, forgetting, summary drift and disaster recovery
  19. ReferenceGlossary
  20. AboutAbout this site and its sources
Practice · 17

Advanced: migration, conflict resolution, forgetting, summary drift and disaster recovery

What happens to saved state when the code changes, when two memories disagree, when a summary is summarized again, and when the database itself is lost.

What you will understand
Old state, new code
What happens to a paused run when the workflow changes under it, and how versioned state keeps it resumable.
Disagreement and decay
Two memories that contradict each other, forgetting as a design, and summaries of summaries.
Losing the database
Backup, restore, recovery point and recovery time, and the tables that must be restored together.

This page deepens the whole site. It assumes the vocabulary of the earlier pages: a run, an attempt, a checkpoint, an idempotency key, the outbox, a memory with provenance and expiry. If a word is new, the glossary link on it leads to its home.

Run run-9d20 for SUB-1009 has been paused for three days. It is case UW-M06, one of this site's own cases: the agent recommended approve, and the run waits at the step underwriter_review for a person to decide. Its last checkpoint holds state_version 8 and a state object written by version 1 of the workflow code.

On the second day the team ships version 2. It adds a step check_flood_zone between check_eligibility and write_recommendation, and it renames the state field roof_assessment to roof. On the third day the underwriter declines, and the worker running version 2 loads the checkpoint to continue.

The new code looks for roof and finds nothing. It sees a run that skipped a step which, for this run, never existed. Nothing in the earlier pages prepared for this. Every mechanism there assumed that the code that wrote the state is the code that reads it. In production that assumption fails every time you deploy while runs are in flight. The rest of this page deals with five problems of that kind, where the system outlives one version of itself.

The agent here is a scripted stand-in, so the results on this page show how the mechanism works, not how good any model is.

1. Saved state meets new code#

A checkpoint is saved execution state that allows a workflow to resume. It is a contract between the code that wrote it and the code that will read it. Versioned state means the checkpoint carries the version of the state's shape, for example "state_schema": 1, so that a reader knows which contract it holds.

State migration is changing saved state from an old shape to a new one. For example, renaming roof_assessment to roof in every checkpoint of every paused run, or doing it as each one is loaded.

There are three designs, and each has a cost.

DesignHow it worksCostGood when
DrainStop starting runs on v1; let them finish; then switchLong-paused runs block the release, sometimes for daysRuns are short
Migrate on readv2 reads state_schema, upgrades a v1 state in memory (an upcaster), continuesEvery shape change needs an upcaster and a test; they accumulateShapes change often, runs are long
Run bothv1 workers finish v1 runs; v2 workers take new runsTwo code versions in production until the last v1 run endsChanges to the step sequence, not only the shape

The rename is a shape change; an upcaster fixes it. The new step is a sequence change, and it is harder. For run-9d20, step 6 already ran under v1 without the flood check. Should v2 run check_flood_zone now, after the recommendation was written? The answer depends on the business, not the code: the recommendation was made without a flood check, and it is the underwriter who decides. A safe default is to finish old runs on the old sequence and record the code version on every checkpoint, so that an auditor can see which sequence a run followed.

Diagram: A paused run meets new code · read the version before reading the state

Read left to right. The checkpoint was written under version 1 and says so. The new worker reads the schema number first. A shape change is upgraded in memory by an upcaster and the run continues under version 2. A change to the sequence of steps sends the run to a version 1 worker, which finishes the old sequence. No checkpoint is read by code that does not understand it.

  1. 1 · Saved under v1
    1. Checkpoint: The paused run's saved state, carrying the version of its shape and of the code that wrote it.
  2. 2 · Read the version
    1. Version check: The worker under version 2 reads the schema number before any field.
  3. 3a · Shape changed
    1. Upcaster: A small function that turns an old shape into the new one when the state is loaded.
  4. 3b · Steps changed
    1. v1 worker: Runs started under v1 finish on the v1 sequence; the code version is recorded for audit.

Read the diagram left to right. A checkpoint written under v1 carries its schema number. The reader checks it. A shape change goes through an upcaster and continues under v2. A sequence change routes the run to a v1 worker that finishes the old sequence. In both paths the checkpoint is never read by code that does not understand it.

For a backend engineer. This is a database schema migration with a twist. A table migration runs once, in a transaction. Saved run state is often a JSON blob in many rows, each read at an unknown future time. It behaves more like a serialized Java object in a long-lived session: change the class, and old serialized instances must still deserialize. The habit that helps is the same one serialization teaches: a version field in the payload and a reader that handles every version still in storage.

Durable workflow engines that replay a run's history to rebuild state are the strictest case: a code change that alters the order of recorded steps can make an old history unreadable by the new code. How a named engine handles this is described from its own documentation on Frameworks.

2. Two memories that disagree#

The memory store of the example holds MEM-0031 in scope broker:BRK-017: "This broker reports roof age from the owner's statement, not from an inspection", written from an underwriter's decision, assigned confidence 0.95. A month later the broker writes: "From now on our roof ages come from inspection reports." The write policy stores that as a broker statement, assigned confidence 0.5. Both are active. They contradict each other.

Conflict resolution is the rule that decides what a reader is told when two memories disagree. There are four common rules.

RuleWhat the reader getsRisk
Latest winsThe broker statementA low-trust source replaces a high-trust one by being newer
Most trusted winsThe underwriter's memoryA true change is ignored until someone with authority confirms it
Both, flaggedBoth, marked as a conflictThe reader, or the underwriter, must resolve it
Supersede on writeThe policy marks the old one superseded when the new one is storedOnly safe when the new source is at least as trusted

For memories that change a recommendation, "both, flagged" is the honest default: the agent cites neither as a reason and lists the conflict for the underwriter, who is the person with authority to settle it. Her decision then becomes a new memory from source underwriter_decision, which may supersede the old one. "Supersede on write" is right only in the narrow case where the new source is at least as trusted as the old: a later underwriter decision may supersede an earlier one; a broker statement may not supersede an underwriter decision.

The write policy enforces this with a uniqueness rule: at most one active memory per scope and subject. A second candidate for broker:BRK-017 and roof_age_source cannot be silently added. It must either supersede the first, if its source is trusted enough, or be stored as a conflict for review.

3. Forgetting as a design#

Forgetting means removing or downgrading memories on purpose. The earlier pages have three forms of it already: expiry (a broker statement stops being returned after 30 days), deletion with a tombstone (on request or by retention), and the recency signal in ranking (an old memory scores lower). They are different tools.

FormWhat it doesWhen it is right
ExpiryThe memory is no longer returned after a dateThe information goes stale on a known schedule
Decay in rankingThe memory is still returned, but ranks lower with ageOlder information is less likely, not less true
SupersessionA newer memory replaces it; the old one stays for auditThe information changed
DeletionThe text is removed; a tombstone remainsThe information must not exist, by request or schedule

The design question is which form each kind of memory gets, and it is decided per kind, not per store. An underwriter decision that set a semantic memory should decay slowly and be superseded explicitly. A broker statement should expire. A preference should live until the person changes it. Episodic memory is rarely forgotten, but it must be read as history once its policy version is old.

A store that never forgets does not become wiser. It becomes slower to search and more likely to return something stale ahead of something current. A store that forgets by accident, when a cache is cleared or a window rolls over, is worse: it loses what was supposed to stay. Forgetting deserves the same deliberate design as writing.

4. Summaries of summaries#

A summary is a lossy copy. History or facts showed what one summary loses. Summary drift is what happens when the summary is itself summarized again, together with new material, round after round.

Take the twelve-message exchange about SUB-1003. Round 1 summarizes turns 1 to 8: "The broker states the roof was replaced in 2019; invoice to follow." Round 2 summarizes that summary plus turns 9 to 12: "Roof replaced 2019; invoice pending." Round 3, a week later: "Roof replaced 2019." In three rounds a claim with a pending proof became a plain fact, with no single step that anyone would call wrong.

Three habits keep drift bounded. Summarize from the original turns, not from the previous summary, whenever the originals are still kept. Keep claims as durable facts with a source type, outside the summary, so that their status cannot be compressed away. And link each summary to the range of turns it covers, so that any statement in it can be checked against its source.

5. Losing the database: backup, restore and disaster recovery#

Disaster recovery is bringing a system back after losing a whole store or site, not one process. For example, the database of the example is lost along with its disk, and the last backup is from 03:00.

Two numbers frame the plan. The recovery point is how far back the restored data is: a recovery point objective (RPO) is the maximum acceptable time between the last recovery point and the failure, which is the data you accept losing. The recovery time objective (RTO) is the maximum acceptable delay before service is back. Both are set by the organization, not by the technology. A backup at 03:00 and a failure at 10:00 means up to seven hours of runs, checkpoints and memories are gone.

The tables of this site do not fail independently, and that is the hard part. Consider what a restore to 03:00 does to run run-7f3a, which finished at 09:40.

TableAfter the restoreConsequence
runs, checkpointsrun-7f3a does not existThe submission looks untouched; a new run starts
recommendationsNo rowThe new run writes a recommendation again
outboxNo rowFine: no message pending
Cockpit inbox (another system, not restored)Holds the 09:41 message for run-7f3aThe underwriter already saw a recommendation the insurer's store no longer has
memoriesMemories written after 03:00 are gone; tombstones written after 03:00 are goneA note deleted at 08:00 on request is back

The last row is the dangerous one. An erasure that was carried out is undone by a restore, silently. The repair is to keep the deletion log outside the main backup cycle, or with a recovery point shorter than the main one, and to replay it after every restore. The same holds for the cockpit's view: the restore plan must say what the systems that were not restored will see, and how the gap is reconciled. Here the reconciliation is to compare the cockpit's inbox with the restored recommendations table and re-import what the cockpit holds, with its original idempotency keys, so that the new run does not write a second recommendation.

Production pitfall. A backup that has never been restored is a hope, not a recovery plan. Restore into a separate environment on a schedule, run the recovery test from Evaluation against it, and check that deleted memories are still deleted.

How the five fit together#

All five problems share one cause: the system lives longer than any single version of its code, its data or its people's intentions. Saved state outlives the code that wrote it. A memory outlives the situation it described. A summary outlives the turns it compressed. A backup outlives the deletions made after it. Each fix makes the version, the source or the time explicit, so that a reader can tell what it holds.

A checklist for a Principal engineer#

  1. Does every checkpoint carry the schema version of its state and the code version that wrote it?
  2. What happens to a paused run during a deploy that changes the step sequence? Is that tested?
  3. Is there at most one active memory per scope and subject, and what happens to the second candidate?
  4. For each kind of memory, which form of forgetting applies?
  5. Are summaries built from originals, and is each linked to the range it covers?
  6. What are the RPO and RTO, who set them, and when was the last restore test?
  7. After a restore, how are deletions re-applied, and how are systems that were not restored reconciled?
Deeper: testing a state migration

The test that matters loads real old states, not freshly made ones. Keep a corpus of checkpoints from every state schema still present in storage, made by running the old code on the cases of the example, and run every upcaster over every one in CI. Then resume each upgraded state under the new code and compare the end state with an uninterrupted run under the new code, the end-state equivalence of Evaluation. An upcaster may be removed only when no stored state of that schema remains, which a query over the checkpoint store can prove.

Deeper: restoring tables to one consistent point

The tables that implement the earlier pages, checkpoints, recommendations, outbox, the idempotency records and the consumer's inbox, are correct only together. If they live in one database and are backed up as one consistent snapshot, a restore keeps them consistent with each other, and the problem moves to systems outside that database. If they live in different stores with different backup times, a restore can produce a recommendation without its outbox row, or an outbox row whose message the consumer's restored inbox no longer remembers, which will then be processed twice. That is a strong argument for keeping the run's state, its effects' records and the outbox in one transactional store, the design this site uses, and treating everything outside it as a consumer that deduplicates.

In one breath#

Production systems outlive their own versions. Saved state needs a schema version and a migration path, and a change to the step sequence is harder than a change to the shape. Two memories that disagree need a rule, usually "both, flagged" for a person, and a uniqueness rule that turns contradictions into visible conflicts. Forgetting is chosen per kind. Summaries are rebuilt from originals. And a restore is part of the design: it must keep the run's tables consistent and re-apply deletions.

The same idea for a coding agent: a long session resumed after the repository's instruction file changed is a migration; two instruction files that contradict each other need a precedence rule; and an old compacted summary of the session is the summary drift of this page.

This is the end of the series of chapters. The Insurance lab shows the whole example in one place, and the Glossary collects every term.