All chapters
  1. Start hereMemory is what you keep. A checkpoint is where you stopped
  2. Memory · 1The stateless request: why anything has to be stored at all
  3. Memory · 2Five kinds of memory: working, episodic, semantic, procedural and preference
  4. Memory · 3Conversation history, summaries, retrieval and durable facts
  5. Memory · 4Files as memory in coding agents: instructions, journals, artifacts and checkpoints
  6. Memory · 5Database stores: tables, JSON, documents, objects and vectors
  7. State · 6Run identity: run_id, thread_id, trace_id, correlation_id and task_id
  8. State · 7State machines, checkpoints, event sourcing and where replay must stop
  9. State · 8Failure and resume: idempotency keys, retry policy and the exactly-once illusion
  10. State · 9The transactional outbox: side effects, compensation and deduplication
  11. Policy · 10Memory write policy: extraction, confidence, provenance, expiry and deletion
  12. Policy · 11Retrieval and ranking: relevance, recency, salience and scope
  13. Policy · 12Shared memory between agents: consistency, isolation, ownership and races
  14. Policy · 13Security: tenant separation, personal data, secrets, consent and retention
  15. Practice · 14Evaluation: memory correctness, stale facts, contamination and recovery tests
  16. Practice · 15Frameworks and standards: what each one stores for you
  17. Practice · 16The insurance lab: one run that fails, resumes and remembers, end to end
  18. Practice · 17Advanced: migration, conflict resolution, forgetting, summary drift and disaster recovery
  19. ReferenceGlossary
  20. AboutAbout this site and its sources
State · 6

Run identity: run_id, thread_id, trace_id, correlation_id and task_id

Six ids that travel with one piece of work and mean six different things. What each one names, who issues it, and what breaks when two of them are merged.

What you will understand
Ten ids, ten meanings
What each id names, who issues it, and how long it lives.
Run, attempt, trace
One run, two attempts, two traces: the three ids that blur most often.
What merging breaks
The concrete failure each wrong merge causes, from a lost resume to a swallowed write.

An underwriter opens the cockpit on a Tuesday and asks a simple question about submission SUB-1004: "What happened with this one?" The platform engineer on call opens the database and finds the answer spread across rows that each carry a different id. The recommendation row says run-7f3a. The checkpoints say run-7f3a too, but they also say attempt 1 and attempt 2. The tracing tool shows two traces, 1d237d5c… and f1cb96bc…, which look like two unrelated executions. The message to the cockpit carries corr-5c1e. The checkpoints are filed under thread-SUB-1004.

Every one of those ids is correct. Each names something different. If the engineer, or the code, treats two of them as the same thing, something breaks: a resume that cannot find its checkpoint, a write that is swallowed as a duplicate, two traces that look like two runs, a memory filed under the wrong scope. This page is the home of every identifier on the site. It names each one, says who issues it and how long it lives, and shows what goes wrong when two are merged. The case is UW-M01, the anchor of this site's own cases.

The agent here is a scripted stand-in, so the results on this page show how the mechanism works, not how good any model is.

Start with the business object#

The first id belongs to the thing the business cares about. The submission id names one quote request: one broker, one building, one request. For example, SUB-1004. Intake issues it when it turns the broker's email into a record, and it lives for years, as long as the insurer keeps the file. Other specifications in this series call it application_id; this site says submission.

Next to it sits the tenant. broker_id, for example BRK-017, names the brokerage that sent the submission. On this site it is the scope that memory is filed under, so that one broker's memories never reach another broker's run. Security explains why that matters.

Neither of these says anything about execution. A submission can be worked on several times: once when it arrives, again when the broker sends a corrected roof age. Each of those is a separate piece of work, and needs its own name.

The run, the thread and the attempt#

A run id names one execution of the workflow for a submission, from its start to a final state. For example, run-7f3a for SUB-1004. The workflow runtime issues it when the run starts. While the run lives, its checkpoints, its recommendation and its outbox message are filed under it; afterwards it is kept for audit.

A thread id names the line of work that checkpoints are filed under. For example, thread-SUB-1004: on this site there is one thread per submission. Several runs can share a thread. If the broker's corrected roof age arrives a week later and a second run starts, it joins the same thread, so that someone looking at the thread sees both executions in order. A thread lives as long as the submission is open.

An attempt number names one process-level try inside a run. For example, attempt 1 of run-7f3a ran two steps and died; attempt 2 resumed from the checkpoint and continued. The runtime issues the number. A resume after a crash, or after a pause for a person, is the next attempt of the same run, never a new run. Attempts last minutes.

The example's schema keeps these apart by construction. The runs table has one row per run with its thread and its correlation id; the attempts table has one row per attempt, keyed by the run and the number. This is from schema.sql:

CREATE TABLE IF NOT EXISTS attempts (
  run_id               TEXT NOT NULL REFERENCES runs(run_id),
  attempt_number       INTEGER NOT NULL,             -- 1, 2
  trace_id             TEXT NOT NULL UNIQUE,         -- 32 hex; one trace per attempt
  status               TEXT NOT NULL CHECK (status IN ('running', 'ok', 'crashed', 'paused')),
  resumed_from_version INTEGER,                      -- the state_version it started from; NULL for a fresh start
  started_at           TEXT NOT NULL,
  ended_at             TEXT,
  ended_reason         TEXT,
  PRIMARY KEY (run_id, attempt_number)
);
Common confusion. A retry is not an attempt. When the provider times out and the worker calls it again half a second later, the process is still alive; that is a call retry inside a step, and the attempt number does not change. Failure and resume draws that line in full.

The trace and the span#

A trace id names the record of what one process did, for the people who diagnose it. For example, 1d237d5cc6e5fd9a1525a7aa3112466d, 32 hexadecimal characters. A span id names one step inside a trace, 16 hexadecimal characters, such as 3b6334209c002d72 for get_roof_assessment in attempt 1. Both are issued by the tracing library. The W3C Trace Context standard defines their length and format; Frameworks summarizes it from the standard's own text.

On this site there is one trace per attempt. A resumed attempt is a new process at a later time, so it gets a new trace id. The run id is what joins the two traces. The example builds trace ids from the run and the attempt, so every reader gets the same ids; this is trace_id_for in store.py:

def trace_id_for(run_id: str, attempt_number: int) -> str:
    """One trace per attempt: 32 hex characters. A resumed attempt gets a new trace_id."""
    return hex_id(f"trace:{run_id}:{attempt_number}", 32)

In production the tracing library generates trace ids at random. Deriving them from the run, as the example does, is a teaching convenience that keeps the pages reproducible; it is not a recommendation.

Why one trace per attempt, and not one per run? A trace is a tree of spans with a start and an end, recorded by one process. Attempt 2 of run-7f3a started a minute after attempt 1 died, in another process that had no access to attempt 1's open spans. Stretching one trace across the crash would mean storing trace state durably and handing it to the next process, which tracing libraries are not built for. Keeping one trace per attempt, and the run id as an attribute on every span, gives a clean trace for each process and a single key, the run id, to find them all.

What this is not. A trace explains a run. A checkpoint resumes it. A trace is evidence for people; it may be sampled or dropped, and no worker resumes from it. Traces, logs and audit events are the subject of observe.sammuthu.com.

The ids that cross systems and agents#

A correlation id names one business request across every system it touches: the broker's email, the run, the outbox message, the cockpit item. For example, corr-5c1e. The first system that sees the request issues it, here intake, and every later system copies it forward. It lives as long as the request. When the underwriter asks "what happened with this one?", the correlation id is the thread that ties the email to the cockpit item, across systems that share no database.

A task id names one unit of work handed to an agent or a worker inside a run. For example, in the multi-agent version of the example, the underwriting agent asks the assessment agent for a roof report and names that request task-roof-01. Whoever delegates the work issues it, and it lives for one delegation. The example runs as one agent, so it has no task ids; they appear where work is handed between agents, and the protocols that carry them are covered on a2a.sammuthu.com.

The ids of state and effects#

Three more ids belong to what a run stores.

A state version is the sequence number of a checkpoint inside one run: 1, 2, 3. For example, state_version 2 of run-7f3a, also written cp-run-7f3a-0002, is the state after the roof assessment. The checkpoint store issues it, and it counts across attempts because it belongs to the run.

An idempotency key names one intended effect: run-7f3a:record_recommendation. It is built from the run and the step, never the attempt. Its home is Failure and resume.

An evidence id names one stored piece of evidence that a reason or a memory points to: EV-ROOF-0412. Its home is Write policy.

All of them, side by side#

The prose prepared the table; here it is in one place.

IdExampleNamesIssued byLives
submission_idSUB-1004One quote requestIntakeYears
broker_idBRK-017The brokerage; the tenant scope of memoryIntakeYears
thread_idthread-SUB-1004The line of work checkpoints are filed underWorkflow runtimeWhile the submission is open
run_idrun-7f3aOne execution of the workflowWorkflow runtimeUntil the run ends; kept for audit
attempt_number1, 2One process-level try inside a runWorkflow runtimeMinutes
trace_id1d237d5c… (32 hex)One attempt's record for diagnosisTracing libraryOne attempt
span_id3b633420… (16 hex)One step inside a traceTracing libraryOne step
correlation_idcorr-5c1eOne business request across systemsThe first system that sees itThe whole request
task_idtask-roof-01One delegated unit of workWhoever delegatesOne delegation
state_version2One checkpoint inside a runCheckpoint storeOne run
idempotency_keyrun-7f3a:record_recommendationOne intended effectThe caller of the writeAs long as a repeat can arrive
evidence_idsEV-ROOF-0412One stored piece of evidenceEvidence storeYears

The diagram shows how they nest for UW-M01.

Diagram: How the ids of UW-M01 nest · from the business request to one step

Read left to right, from the outside in. The correlation id wraps everything one broker's request causes. Inside it, the submission and its thread, then one run. The run has two attempts, each with its own trace, and each trace has one span per step. Checkpoints and the idempotency key belong to the run, which is why they survive the crash.

  1. Across systems
    1. correlation_id: One business request across the email, the run, the outbox message and the cockpit.
  2. The business object
    1. submission_id: One quote request. Lives for years.
    2. thread_id: The line of work checkpoints are filed under; several runs can share it.
  3. One execution
    1. run_id: One execution of the workflow. Owns the checkpoints (state_version) and the idempotency key.
  4. Process tries
    1. attempt 1: Ran two steps and crashed. Its trace records them.
    2. attempt 2: Resumed from state_version 2 in a new process, with a new trace.

Read it from the outside in. The correlation id wraps everything that one broker's request causes. Inside it, the submission and its thread. Inside the thread, one run. Inside the run, two attempts, each with its own trace. Inside each trace, one span per step. The checkpoints and the idempotency key belong to the run, not to any attempt, which is why they survive the crash.

Explore the run table#

The lab below loads the run tables of the example. Click any id, such as a trace_id or the run_id, and every row that carries it lights up, with a note on what the id names, who issues it and how long it lives.

AttemptTrace id (first 8)StatusFirst and last state version
11d237d5ccrashed1 to 2
2f1cb96bcpaused (at underwriter_review)3 to 9

Two attempts, two traces, one run. The state versions continue across the crash: attempt 2 started where attempt 1's last checkpoint ended.

What breaks when two ids are merged#

Each merge below is a mistake teams make, shown with its consequence in the example.

MergedWhat goes wrong
run_id and trace_idAttempt 2 gets a new trace id, so it cannot find the run's checkpoints and starts a new run from step 1: a second provider call
run_id and attempt_number (a new run per resume)The idempotency key changes on resume, so the recommendation is written twice
attempt_number in the idempotency keySame failure: two recommendation rows after a crash between the write and its checkpoint (UW-M04)
thread_id and run_idA second run for the same submission overwrites or continues the first run's checkpoints
correlation_id and run_idThe cockpit item cannot be traced back to the email, or a second run for the same request looks like a different request
submission_id and broker_id as memory scopeEither memory is too narrow to help the next submission, or it leaks across brokers
Call retry and attemptDashboards count timeouts as crashes, and resume logic runs on a live process

The common thread: an id must name exactly one thing with one lifetime. When a value is reused for a second purpose, whichever of the two lives shorter decides when the other breaks.

For a backend engineer. You already keep these apart in a service: an order id, a request id per HTTP call, a trace id from your tracing library, a correlation id in a message header, a database row version for optimistic locking. A Kafka consumer that reprocesses a message after a crash keeps the message key but gets a new trace. Agents add two ids that services rarely need, the attempt number and the state version, because an agent's unit of work outlives a process.

Framework implementation#

Frameworks name these things differently: some file checkpoints under a thread and call each execution a run, some call the whole durable execution a workflow with its own id, some give each retried step its own attempt count. What each named framework's ids mean, from its own documentation, is on Frameworks. When adopting one, map each of its ids to a row of the table above, and look for any row that has no counterpart.

A checklist for Monday#

  1. List every id your agent system writes. For each, write the row of the table: what it names, who issues it, how long it lives.
  2. Does a resumed run keep its run id and get a new attempt number? Check the database after a forced crash.
  3. Is any idempotency key built from a value that changes on resume?
  4. Can you get from a cockpit item, or its equivalent, back to the original request with one id?
  5. Does every span carry the run id, so that the traces of all attempts can be found together?
Advanced: who may issue an id, and why that matters

Each id has one issuer, and the rule that only that issuer creates it is what makes it trustworthy. If a worker could invent a run id, two workers that both think a run is new would create two runs for one submission. In the example the run id is created in one place, the function start_run in workflow.py. A production design adds a unique constraint on "one open run per submission", so that a second run cannot start while one is open.

Ids that cross a trust boundary need more care. A correlation id copied from the broker's email header is attacker-controlled input: it must be validated, and it must never be used as a key into another tenant's data. Trace ids carry their own rule; the W3C Trace Context standard says vendors must not put personally identifiable or sensitive information in the trace headers, which is one more reason not to derive them from business ids in production.

Lifetimes also bound storage. The attempts table grows with every crash, the events table with every step. Retention follows the longest-lived id that refers to a row: an attempt row can go when its run is archived, but an evidence row lives as long as any decision cites it.

In one breath#

Each id names one thing with one issuer and one lifetime: the submission, the broker, the thread, the run, the attempt, the trace and span, the correlation across systems, the delegated task, the state version, the intended effect and the evidence. One run can have several attempts and, on this site, one trace per attempt; the run id joins them, and the checkpoints and idempotency keys belong to the run. Merge two ids and the shorter-lived one breaks the other.

The same idea for a coding agent: the branch is the thread, a session is an attempt, the pull request number is the correlation id that ties the issue, the commits and the CI runs together.

Every row of the run can now be found and joined. The next page builds the record that the run id points to: the named states of a run, saved after each step, and the line replay must not cross. Checkpoints.