All chapters
  1. Start hereMemory is what you keep. A checkpoint is where you stopped
  2. Memory · 1The stateless request: why anything has to be stored at all
  3. Memory · 2Five kinds of memory: working, episodic, semantic, procedural and preference
  4. Memory · 3Conversation history, summaries, retrieval and durable facts
  5. Memory · 4Files as memory in coding agents: instructions, journals, artifacts and checkpoints
  6. Memory · 5Database stores: tables, JSON, documents, objects and vectors
  7. State · 6Run identity: run_id, thread_id, trace_id, correlation_id and task_id
  8. State · 7State machines, checkpoints, event sourcing and where replay must stop
  9. State · 8Failure and resume: idempotency keys, retry policy and the exactly-once illusion
  10. State · 9The transactional outbox: side effects, compensation and deduplication
  11. Policy · 10Memory write policy: extraction, confidence, provenance, expiry and deletion
  12. Policy · 11Retrieval and ranking: relevance, recency, salience and scope
  13. Policy · 12Shared memory between agents: consistency, isolation, ownership and races
  14. Policy · 13Security: tenant separation, personal data, secrets, consent and retention
  15. Practice · 14Evaluation: memory correctness, stale facts, contamination and recovery tests
  16. Practice · 15Frameworks and standards: what each one stores for you
  17. Practice · 16The insurance lab: one run that fails, resumes and remembers, end to end
  18. Practice · 17Advanced: migration, conflict resolution, forgetting, summary drift and disaster recovery
  19. ReferenceGlossary
  20. AboutAbout this site and its sources
Practice · 14

Evaluation: memory correctness, stale facts, contamination and recovery tests

How to test what was stored and what came back: wrong memories, stale ones, memories from the wrong scope, and a crash at every step followed by a resume.

What you will understand
Testing memory
Write precision, read correctness, stale facts and contamination: four questions with four tests.
Testing recovery
Crash after every step, resume, and compare the end state with a run that never crashed.
What a pass misses
Why an equal end state is necessary, not sufficient, and what else to count.

Every page so far added a mechanism: checkpoints, idempotency keys, the outbox, a write policy, a scope filter. Each one was shown working on one case. The platform team now wants to change the workflow: a new step, a different retry policy, a new memory library. How do they know that nothing they built still holds?

Not by reading the code. The failures of this site are quiet: a second recommendation row, a lost message, a memory from the wrong broker, a claim that outlived its expiry. None of them raises an error. Each is visible only if something checks the end state after the thing went wrong. This page turns the earlier pages into tests: four for memory, and one grid for recovery that crashes the worker at every point and checks the result.

The agent here is a scripted stand-in, so the results on this page show how the mechanism works, not how good any model is. All counts below are from the example's runner and are synthetic.

What this is not. Measuring how good the agent's recommendations are, with golden sets, graders, judges and release gates, is evaluation in the broad sense and has its own site: eval.sammuthu.com. This page tests only two things: that the store holds and returns the right memories, and that a crashed run recovers to the right state.

Four questions about memory#

Memory correctness is the property that the store holds what it should and returns what it should, to the right reader, at the right time. It breaks down into four questions, each with its own test.

The first question is whether the right candidates were stored. Write precision is the share of stored memories that a person would agree should have been stored. For example, if the remember step stores four memories after a run and a reviewer agrees with three, write precision is 3 of 4. It is measured against a small set of finished runs for which a person has written down what should have been kept. Its partner, recall, is the share of the memories that should have been kept that were. The example's tests check the policy side of this directly: the refusals of UW-M08 and the stored broker statement of UW-M09 are asserted outcomes, not judgments.

The second question is whether a read returns the right memory. For a set of questions with known answers, does the top-k contain the memory a person expects? This is the test that tunes the ranking weights on Retrieval.

The third question is about time. A stale fact is a memory that was true when written and is no longer true, or no longer valid, when read. For example, UW-M09's broker statement after its expiry, or UW-M07's approval under policy version 2025.4 read under 2026.2. The test moves the clock or the policy version and asserts that the memory is no longer returned, or is returned only as history.

The fourth question is about scope. Contamination is a memory reaching a reader outside its scope: another broker, another submission, another person. For example, MEM-0003 of broker BRK-022 in a read for BRK-017. The test plants a perfect match in a foreign scope and asserts that it never comes back. That is case UW-M11.

QuestionTestCase in the examplePasses when
Were the right things stored?Write precision and recall against reviewed runsUW-M08, UW-M09Refusals refused, accepted candidates stored with the right source, confidence and expiry
Does a read return the right memory?Questions with expected memoriesUW-M11The expected memory is in the top k
Are stale facts kept out?Move the clock or the policy versionUW-M09, UW-M07Expired memories are not returned; old-policy episodes come back only as history
Does memory stay in its scope?Plant a foreign perfect matchUW-M11The foreign memory is never a candidate

A deletion test belongs to the same family: delete a memory and assert that no read path returns it, including derived summaries. That is UW-M12, and Security explains why the summary matters.

The recovery test#

The second family is about state. A recovery test crashes the worker at a chosen point, lets the run resume, and checks the result. The check is end-state equivalence: the end state of the crashed-and-resumed run must equal the end state of the same run with no crash. For example, run-7f3a without a crash ends with one recommendation row, one cockpit item and the memories its remember step writes; after a crash at any point, it must end with exactly the same.

The example defines the end state as four counts, and the recovery test is a real run against a fresh database each time. This is from recovery.py:

END_STATE_FIELDS = ("recommendations", "cockpit_items", "decisions", "memories")
PHASES = ("before_checkpoint", "after_checkpoint")
MEMORIES_PER_RUN = 3      # what step 10 stores for SUB-1004 in this example (see the README)

The grid crosses four things: the step after which the worker dies (ten steps), whether the crash comes before or after that step's checkpoint is saved (two phases), whether checkpoints are kept at all, and whether idempotency keys are used. That is 10 × 2 × 2 × 2 = 80 recovery tests, each a separate run. The runner prints the summary at the end of --all:

python run_memory.py --all
Recovery tests (SUB-1004, crash after each step × before/after its checkpoint):
  checkpoints on  · keys on  · 20 of 20 pass · most provider calls 2
  checkpoints on  · keys off · 17 of 20 pass · most provider calls 2 · failing: record_recommendation (before), notify_cockpit (before), remember (before)
  checkpoints off · keys on  · 20 of 20 pass · most provider calls 2
  checkpoints off · keys off · 12 of 20 pass · most provider calls 2 · failing: record_recommendation (before), record_recommendation (after), notify_cockpit (before), notify_cockpit (after), underwriter_review (before), underwriter_review (after), remember (before), remember (after)

Read the four lines as four designs.

With checkpoints and keys, all 20 crash points recover to the same end state. With checkpoints and no keys, three crash points fail, and they are exactly the ones in the gap that Failure and resume described: a step with a side effect finished, the worker died before its checkpoint, and the step ran again without a key to stop the repeat. Without checkpoints but with keys, all 20 pass, because the whole run starts over and every write is deduplicated. Without either, eight fail.

Explore the same grid in the lab. Each cell is one recovery test; click a cell to see what differs from the uninterrupted run.

CheckpointsKeysPassing recovery testsFailing crash points
OnOn20 of 20none
OnOff17 of 20record_recommendation, notify_cockpit, remember, each before its checkpoint
OffOn20 of 20none
OffOff12 of 20record_recommendation, notify_cockpit, underwriter_review, remember, before and after

The diagram shows how one recovery test is built.

Diagram: One recovery test · crash, resume, compare with a run that never crashed

Read left to right. A reference run without a crash gives the expected end state. Each test starts from a fresh store, injects one crash at a named point, lets the run resume until it rests, and counts the same four things. Equal counts pass. A difference names the side effect that repeated. Counting outside calls as well shows what an equal end state hides.

  1. 1 · Reference
    1. Uninterrupted run: run-7f3a for SUB-1004 with no crash: recommendations, cockpit items, decisions and memories counted once.
  2. 2 · Inject
    1. Crash: A fresh store, then the worker dies at one named point.
  3. 3 · Recover
    1. Resume: The next attempt starts from the last checkpoint, or from the start if checkpoints are off.
  4. 4 · Compare
    1. End state and costs: Equal counts pass. Provider calls and repeated steps are counted too, because an equal end state can hide a second paid call.

Read it left to right. A reference run without a crash produces the expected end state once. Each test starts from a fresh store, injects one crash, lets the run resume until it rests, and counts the same four things. Equal counts pass. The difference, when there is one, names the side effect that repeated.

What a pass does not tell you#

Look again at the third design: no checkpoints, keys on, 20 of 20 pass. By the end-state test it is as good as the first design. It is not. Without checkpoints every crash restarts the run from step 1, so a crash after the roof assessment calls the provider a second time. The end state is equal; the cost is not, and the second call might even return a different roof age. The runner reports "most provider calls 2" on every line for this reason: in the first design, a crash after the provider call but before its checkpoint also repeats the call, once.

So end-state equivalence is necessary and not sufficient. A recovery test should also count what the run did on the way: provider calls, steps executed twice, messages sent before deduplication. The example records the executions per step for each test for exactly this purpose. A design that passes on end state and doubles the outside calls is correct and expensive; whether that is acceptable is a decision, and the test should make it visible.

Two more limits. The grid crashes at step boundaries, before or after a checkpoint. A real crash can fall in the middle of a step, between a provider's response and the worker's write of it. And the grid has one case, SUB-1004. Both are reasons to add crash points inside steps and to run the grid on several cases, not reasons to trust 20 of 20 more than it deserves.

For a backend engineer. This is chaos testing at the size of one function call. Instead of killing pods in a cluster and watching dashboards, the test kills the worker at a named point and asserts on the database. It is cheap enough to run on every change, which a cluster-level experiment is not. Teams that already run consumer tests with a crash between processing and the offset commit are doing the same thing for a Kafka consumer.

Running the tests on every change#

The example's test_memory.py holds 32 unit tests, including each case's expectation on a fresh store, all twelve cases in one shared store, and a check that the pure model of the recovery grid equals the 80 real runs. On the machine used for this page all 32 ran in under a second ("Ran 32 tests in 0.245s"), so they can run on every change. A change that breaks idempotency in one step fails a named test with a named crash point, which is far easier to fix than a duplicate found by an underwriter.

python -m unittest -v

The output lists each test by name. Run it in a scratch copy of the example, as the README says, because the runner writes into out/.

Framework implementation#

Some workflow engines ship test helpers that replay a recorded history against new code, or run a workflow with time skipped. Those help with determinism and long waits; they do not decide what your end state is. What a named framework offers for testing is on Frameworks. The end-state definition and the crash grid stay yours.

A checklist for Monday#

  1. Write down your run's end state as a few counts: rows written, messages consumed, decisions, memories.
  2. Build one recovery test: crash after one step, resume, compare with an uninterrupted run.
  3. Extend it to every step, before and after its checkpoint. How many pass?
  4. Count outside calls in each test, not only the end state.
  5. Plant a perfect match in a foreign scope and an expired memory in your own. Neither may come back.
  6. Pick ten finished runs, write down what should have been remembered, and measure write precision.
Advanced: measuring write precision without fooling yourself

Write precision needs a person's judgment of what should have been stored, which makes it a labelling task. Keep the label set small and stable: ten to twenty finished runs, chosen to cover the source types and the refusal reasons, each with a written list of expected memories. Two reviewers should label a few runs independently; where they disagree, the write policy is ambiguous, and the fix is a clearer rule, not a vote.

Precision and recall pull against each other. A strict policy stores little and is precise; a loose one remembers more and stores noise. Which side to err on depends on what a wrong memory costs. In underwriting a wrong memory can change a recommendation, so precision matters more than recall, and every refusal is cheap: the information is still in the run's records. Measurement of this kind, with intervals and sample sizes, is covered in depth on eval.sammuthu.com.

Advanced: growing the recovery grid

The example's grid is small because the workflow has ten steps. Real workflows have branches, loops and several agents. Three habits keep the grid useful as it grows. Generate crash points from the workflow definition rather than listing them by hand, so that a new step gets tests automatically. Add crash points inside steps that have more than one side effect. And add two crashes in one run, for example a crash and then a crash during the resume, which catches resume code that is not itself idempotent. The number of tests grows quickly; most of them run in milliseconds against an in-memory database, as the example's do, so the limit is usually the time to read failures, not the time to run them.

In one breath#

Memory needs four tests: were the right things stored, does a read return the right memory, are stale facts kept out, and does memory stay in its scope. Recovery needs a grid: crash after every step, before and after its checkpoint, resume, and compare the end state with an uninterrupted run. An equal end state is necessary and not sufficient; count the outside calls and repeated steps too.

The same idea for a coding agent: interrupt a session after each step of a scripted task, resume from the journal, and diff the repository against an uninterrupted run; a duplicated migration or a second pull-request comment is the failure.

The tests now say whether the mechanisms hold. The next page names the frameworks and standards that implement them, and what each one stores for you. Frameworks.