Agent Memory in Production: What Holds Up and What Doesn't

Every agent that survives past a single turn has a memory problem, and most teams solve it three times before admitting which of the three solutions they actually needed. Transcript replay, structured state files, and retrieval over a vector store are not tiers of the same idea. They fail at different speeds, and only one of those failure modes makes noise.

Transcript replay is honest until it isn’t

Replay means keeping the full message list and resending it every turn. It is the default because it needs no design: the conversation is the state. Debugging is close to free, because the exact input the model saw is the thing you stored. If an answer was wrong, you can point at the turn where the context went sideways.

Two things end this. The first is the context window. You will hit a context_length_exceeded error, which is the good outcome — loud, immediate, reproducible. The second is prompt size creeping upward every turn, which is not a failure but a tax you pay on every request for the life of the session.

Truncation is where people get hurt. Dropping the oldest turns is the obvious fix, and it silently deletes the decision the agent made forty turns ago and now contradicts. The usual band-aid is a model-generated summary of the dropped span, stored as a message. That works until you need to know whether the summary is still true, at which point you are maintaining an unversioned, unverifiable artifact inside your context. This is the same class of problem context engineering describes: memory and prompt layout are one decision, not two.

Replay is right for short-lived work — a support reply, a review pass, a single research question. It is wrong for anything with a status that must survive a process restart.

Structured state files are the boring default

A state file is a typed object you own: task, status, decisions, open questions. You write it to disk or a Postgres row and re-inject it at the top of every prompt. The model reads it; the model does not own it. Your code validates it.

This is the approach that survives production, mostly because it fails visibly. If the agent never records a decision, the field is missing and a schema check catches it. If a decision is wrong, you diff the file. There is no embedding, no similarity threshold, no ranking to reason about.

The cost is that the schema is a product decision, not a technical one. Too few fields and the agent re-derives things it already settled. Too many and the model starts editing fields it does not understand.

{
  "schema_version": 3,
  "task": {
    "goal": "migrate billing webhook from v1 to v2",
    "status": "blocked",
    "blocked_on": "staging credentials"
  },
  "decisions": [
    {"id": "d-014", "choice": "keep idempotency key in header",
     "reason": "v2 spec", "source_turn": 41}
  ],
  "open_questions": ["does v2 retry on 429?"],
  "artifacts": {"diff": "/tmp/billing.patch"}
}

Version the schema from day one and never let the model write fields without a validator. schema_version lets you migrate old sessions instead of discarding them. source_turn lets you trace a decision back to the conversation that produced it, which is the one property a vector store cannot give you cheaply.

Then build the context in code, not in the prompt. The wider pattern of keeping orchestration out of the model is covered in the ultimate AI automation guide.

MAX_CONTEXT_TOKENS = 12_000

def build_context(state, turns, budget=MAX_CONTEXT_TOKENS):
    head = [SYSTEM_PROMPT, render_state(state)]
    used = sum(count_tokens(t) for t in head)   # count_tokens via tiktoken
    tail = []
    for turn in reversed(turns):
        if used + count_tokens(turn) > budget:
            break
        tail.insert(0, turn)
        used += count_tokens(turn)
    dropped = len(turns) - len(tail)
    if dropped:
        head.append(f"[{dropped} earlier turns elided; see state file]")
    return head + tail

The state file is a small fixed cost. The replay window is the variable one. Keeping them separate is what makes failures legible: a bad answer is either a stale field or a missing turn, and you can tell which.

Vector retrieval degrades quietly

Retrieval over a vector store looks the most like memory and behaves the least like it. You embed past turns, notes, or documents; at query time you embed the request, pull the top-k nearest chunks, and paste them in.

Nearest is not relevant, and nothing errors when it is wrong. Retrieval returns chunks semantically close to the query. It has no concept of which facts are current, which were superseded, or which contradict each other. The agent mixes a decision from March with its replacement from June, and the output is fluent and wrong. This is the most expensive failure mode because it does not page anyone.

Retrieval is genuinely good at one job: searching a corpus that does not change — documentation, tickets, transcripts you never edit. It is a poor primary memory for an agent that rewrites its own understanding over time.

If you must use it as memory, store metadata next to the vector and filter before you rank. Scope by session, tag by status, and never let a superseded chunk enter the candidate set.

# eval/cases/memory.yaml
- id: recalls-prior-decision
  setup: fixtures/state_v3.json
  turns:
    - "which idempotency approach did we settle on?"
  assert:
    contains: "header"
    must_not_contain: "I don't have that information"

What to measure before you trust any of them

Do not measure vibes. Instrument four things and keep the raw traces.

The eval set is the part people skip. Ten to twenty cases, each a question whose answer you already know, run after every prompt or schema change. The config above is enough; it is not a benchmark suite and does not need to be. What it catches is the regression you introduce on a Tuesday while editing an unrelated instruction.

Two rules that save hours: log the full rendered context for every call, and record which memory source each fact came from. When an agent goes wrong, you want one query to answer whether the fact was absent, stale, or retrieved and then ignored.

What is still unsolved

Cross-session identity. There is no reliable, cheap way for an agent to know that a task from last week and one from this week are the same task, so state files fragment and retrieval returns both.

Contradiction resolution. Nothing in a vector store tells you which of two mutually exclusive facts is newer. Timestamps help only if every write has one, and even then you must decide whether newer means better.

Forgetting. There is no principled policy for what to drop. Replay forgets by truncation, state files forget by never writing the field, retrieval forgets by falling out of top-k. All three are accidental.

Evaluation. No public benchmark tells you whether your memory layer works on your data. You build the small eval set yourself, and it will be domain-specific by definition. The approaches that hold up in production are the ones where a wrong answer traces back to a specific field or a specific missing turn.

What to do next

1. Write down which memory approach your agent uses today, and name the first failure mode you would see if it broke. If you cannot name one, that is the finding.
2. Move anything that must survive a restart into a typed state file with a schema_version and a validator. Leave the transcript as a log, not as state.
3. Build a five-case eval file this week and run it after every prompt edit. Five cases beat zero cases by a wide margin.
4. Add per-request logging of the rendered context and the source of each fact. One extra line in your trace is usually enough.
5. If you are still assembling the pieces, the AI Developer Stack Bundle ($79, one-time) bundles a prompt library, agent framework, deployment tooling and tutorials, pre-configured to work together, which removes some integration work before you start measuring your own memory design.

Get AI Developer Stack Bundle

AI Developer Stack Bundle — $79, one-time payment, instant download. See the full breakdown on the review page.

Related reading

• The Ultimate Guide to AI Automation 2026
• How I Built a 10-Product Digital Business
• Building an AI Content Factory