Agent Memory Systems

An LLM forgets everything between calls. "Memory" is the set of engineering choices that decide what an agent carries forward: inside one long task (working memory and context management) and across sessions, users and months (long-term memory). Interviewers ask about it because it sits where product, data engineering, retrieval, security and cost all meet. A good answer shows you can design the write path, the read path, the storage and the forgetting policy, and that you know how to evaluate and secure the result, rather than just naming Mem0 or MemGPT.

1. Why agents need memory

A transformer forward pass is a pure function of its input tokens. Nothing persists between API calls except what you send back in. A chat app that "remembers" earlier turns is just re-sending them. So memory in an LLM system is never a property of the model alone. It is a property of the harness around it: what you store, where, and what you put back into the prompt next time.

Four pressures drive the need for something more than "re-send everything":

One more framing helps in interviews: memory vs RAG. Classic RAG retrieves from a mostly static corpus that someone else wrote (docs, wiki). Memory retrieves from a corpus the agent itself writes, continuously, from interactions. That changes almost everything about the problem: the corpus is tiny per user but highly dynamic; facts go stale and contradict each other; time matters; the write path (what to store, how to dedup, how to resolve conflicts) is as hard as the read path; and the data is personal, so deletion and isolation are hard requirements. Retrieval mechanics (embeddings, hybrid search, reranking) carry over from the retrieval page.

2. Taxonomies of memory

2.1 The cognitive-science split (CoALA)

The most-cited framing comes from cognitive architectures. CoALA (Cognitive Architectures for Language Agents) describes an agent as an LLM plus modular memories plus a structured action space Sumers+ 2023. It distinguishes:

CoALA also defines internal actions on memory: retrieval (long-term into working), reasoning (working into working), and learning (writing to any long-term memory, including updating procedural memory) Sumers+ 2023. That vocabulary maps directly onto the read path, the scratchpad, and the write path in an engineering design.

2.2 The engineering split

In system design you'll usually be judged on the engineering axes, not the cognitive labels:

3. Short-term and working memory management

Before you add any database, the first memory problem in every agent is keeping one long conversation or task inside the window without losing what matters. This is "context management" or "context engineering". Here are the techniques, from simplest to most sophisticated.

3.1 Buffers, windows and token budgets

• Full buffer. Send the whole history. Correct until it doesn't fit; cost grows quadratically over a conversation, because turn \(n\) re-sends all \(n-1\) previous turns. Total tokens processed over \(N\) turns of \(t\) tokens each is about \(t \cdot N(N+1)/2\). Prompt caching cuts the price of the repeated prefix but not the window limit.
• Sliding window (last \(k\) turns). Simple and cache-unfriendly (the prefix shifts every turn, so a cached prefix is invalidated unless you drop in large chunks). It forgets the user's opening goal, which is often the most important message.
• Token-budgeted truncation. Allocate a budget per section, e.g. system prompt + tools (fixed), pinned items (task statement, user profile), retrieved memory (≤ \(B_m\) tokens), recent turns (fill the remainder, newest first), and reserve output tokens. Always count with the model's real tokenizer. Truncate whole messages and keep tool call/result pairs together, because most APIs reject an orphaned tool result.
• Pinning. Always keep the first user message (the goal), any explicit constraints, and the latest state. Drop the middle first, which is also where the model attends least.

3.2 Rolling summarisation and compaction

Rolling summary: keep the last \(k\) turns verbatim plus a running summary of everything older. When the buffer exceeds a threshold, summarise the oldest chunk and merge it into the summary. MemGPT does this with a "recursive summary" of evicted messages Packer+ 2023.

Compaction is the agent-era name for the same idea applied when the context nears the limit: replace older turns with a high-fidelity summary and continue Anthropic 2025. Model providers now offer it as an API feature. Anthropic's API, for example, can compact on demand or automatically at a token threshold, optionally keeping recent turns verbatim Anthropic docs.

The craft is in what the summary must keep. A generic "summarise this conversation" prompt loses exactly the details an agent needs next. A good compaction prompt asks for:

3.3 Scratchpads and agent-maintained notes files

The most effective pattern for long-horizon agents in 2025–26 is surprisingly low-tech: let the agent keep notes in files outside the context window and read them back when needed. Anthropic describes this as structured note-taking: an agent maintains a NOTES.md or to-do list to track progress across many tool calls and context resets Anthropic 2025. For multi-session coding, their long-running harness uses an initializer session that creates a progress file (claude-progress.txt), a JSON feature list with every feature initially marked failing, an init script, and git commits. Each later session starts by reading these and ends by updating them Anthropic 2025.

Why it works so well:

• Agent-curated salience. The model writes down what it thinks it will need, at the moment it understands why. That beats a summariser guessing after the fact.
• Survives resets. Notes outlive compaction, crashes and new sessions. Anthropic's memory-tool system prompt literally tells the model to "assume interruption" Anthropic docs.
• Models are trained on file tools. Reading, grepping and editing files is in-distribution for coding-trained models. Letta reported an agent with plain file tools scoring 74.0% on LoCoMo with GPT-4o-mini, above the figure Mem0 reported for its graph variant, and concluded that how the agent manages context matters more than the retrieval mechanism Letta 2025. Treat that as a vendor result, not settled science.
• Inspectable and editable by humans, which matters for trust and debugging.

JSON is better than Markdown for state the agent must not casually rewrite (Anthropic found the model was less likely to inappropriately edit a JSON feature list) Anthropic 2025. Markdown is better for free-form notes.

3.4 Tool-output pruning and clearing

In agentic loops, tool results are usually the biggest consumer of context: a file read, a web page, a SQL result. Once the agent has acted on a result, the raw bytes are rarely needed again. Tool-result clearing replaces old results with a placeholder ("[result cleared; re-run tool if needed]") while keeping the call itself, so the agent knows what it did. Anthropic's API exposes this as a context-editing strategy with a trigger threshold (default 100k input tokens), a number of recent tool uses to keep (default 3), an exclude list of tools, and a minimum amount to clear per activation so the cache invalidation is worth it Anthropic docs. When combined with a memory tool, the model gets a warning before clearing so it can save anything important to memory first.

Related tactics: return references rather than payloads (file path + line range, a query, a URL) so the agent can re-fetch just-in-time Anthropic 2025; have tools truncate and paginate by default; and push heavy exploration into sub-agents with their own clean windows that return condensed summaries (on the order of 1–2k tokens) to the orchestrator.

4. Long-term memory architectures

Six recurring designs. Real systems mix them: Letta combines tiered memory with files, Zep combines episodes with a graph, Mem0 combines fact extraction with hybrid retrieval. Learn each as a mechanism with a characteristic failure mode.

4.1 Vector-store memory (embed and retrieve past interactions)

The baseline. Chunk past conversations (per turn, per exchange, or per session), embed each chunk, store it with metadata (user_id, session_id, timestamp), and at query time embed the current message, retrieve the top-\(k\) similar chunks, and inject them. It is RAG where the corpus is your own history.

• Write cost: one embedding call per chunk; no LLM. Cheap and lossless (the raw text is kept).
• Read cost: one embedding + an ANN query (milliseconds). Injected chunks can be verbose.
• Failure modes: no notion of truth over time (both "I live in London" and "I moved to Berlin" are retrieved, and the model may pick the wrong one); similarity ≠ relevance (a query about "my flight" retrieves every travel chat); granularity issues (a turn-level chunk lacks context, a session-level chunk dilutes the embedding); temporal questions ("what did I say last week?") fail because embeddings don't encode time.

LongMemEval's authors found three cheap upgrades to this baseline help: decompose sessions into finer-grained units, expand the index keys with extracted facts (index a chunk under the facts it contains, not just its raw text), and use time-aware query expansion for temporal questions Wu+ 2024. Adding BM25 alongside vectors (hybrid search) also helps a lot for names, IDs and rare terms.

4.2 Extracted-fact memory (Mem0-style)

Instead of storing raw text, an LLM extracts salient facts from each new exchange and stores those as small, atomic memories ("User is allergic to peanuts"). The original Mem0 paper describes a two-phase pipeline Chhikara+ 2025:

1. Extraction: given the new message pair plus context (a conversation summary and the last \(m=10\) messages), the LLM proposes candidate facts.
2. Update: for each candidate, retrieve the top \(s=10\) most similar existing memories, and ask the LLM (via a tool call) to choose one operation: ADD (new fact), UPDATE (augment or refine an existing one), DELETE (the new fact contradicts an old one), or NOOP (already known).

The paper reported, on LoCoMo, a 26% relative improvement on an LLM-as-judge metric over OpenAI's memory, 91% lower p95 latency than full-context (about 1.4 s vs 17 s), and over 90% fewer tokens; memories averaged about 7k tokens per conversation vs about 26k for the full conversation Chhikara+ 2025. A graph variant (Mem0^g) adds entity and relation extraction and did better on temporal questions.

4.3 Knowledge-graph and temporal memory (Zep / Graphiti)

Facts are not independent: people work at companies, projects have owners, preferences change. A temporal knowledge graph stores entities as nodes and facts as edges (subject, predicate, object), and gives every edge a validity interval. Zep's engine, Graphiti, is the reference design Rasmussen+ 2025:

• Three tiers. An episode subgraph holds the raw messages (non-lossy, the provenance); a semantic entity subgraph holds extracted entities and relation edges, linked back to episodes; a community subgraph clusters strongly connected entities with summaries.
• Bi-temporal model. Each fact carries two timelines. The valid-time timeline \(T\) (\(t_\text{valid}, t_\text{invalid}\)) says when the fact was true in the world. The transaction-time timeline \(T'\) (\(t'_\text{created}, t'_\text{expired}\)) says when the system learned or retired it. This is the same idea as bi-temporal tables in databases.
• Edge invalidation instead of deletion. When a new fact contradicts an existing, temporally overlapping one, the old edge's \(t_\text{invalid}\) is set to the new edge's \(t_\text{valid}\). You can ask both "where does Alice work now?" and "where did Alice work in March?"
• Hybrid retrieval. Cosine similarity, BM25 full-text and breadth-first graph traversal run in parallel, then get reranked (RRF, MMR, graph distance, episode-mention frequency, or a cross-encoder).

Zep reported 94.8% vs MemGPT's 93.4% on the older DMR benchmark (which the paper itself criticises as too easy), and on LongMemEval up to 18.5% accuracy improvement with about 90% lower latency, using about 1.6k context tokens instead of about 115k for full context Rasmussen+ 2025. Open-source Graphiti runs on Neo4j, FalkorDB or Amazon Neptune Graphiti repo.

In the example, on 3 June 2026 Alice says "I started at Globex at the start of May". The system learns it (transaction time 06-03) and back-dates validity to 05-01. The Acme edge is invalidated, not deleted. "Where did Alice work in April 2026?" still answers correctly.

4.4 OS-style hierarchical memory (MemGPT / Letta)

MemGPT treats the LLM like a CPU with a small, fast main memory (the context window) and large, slow external storage, and lets the LLM itself page data in and out with function calls ("virtual context management") Packer+ 2023.

Key mechanics from the paper Packer+ 2023:

• Main context = read-only system instructions + a fixed-size, LLM-editable working context + a FIFO message queue whose evicted messages are replaced with a recursive summary.
• External context = recall storage (the full message history) and archival storage (an arbitrary read/write text database, typically vector-indexed).
• Queue manager issues a "memory pressure" warning at about 70% of the window so the LLM can save important things, and at the limit flushes about half the queue into recall storage plus summary.
• Self-editing memory: the LLM decides what to write to working context or archival storage by calling functions. Function chaining via a heartbeat flag lets it make several memory calls before replying.

Letta (the company built around MemGPT) generalised the working context into memory blocks: labelled, size-limited strings (default persona and human) that always sit in the context, each with a description that tells the agent what belongs there Letta docs. Blocks can be shared between agents (useful for multi-agent shared state). Letta later added "sleep-time" agents that rewrite memory in the background between interactions Lin+ 2025.

4.5 Reflection-based memory (Generative Agents)

Park et al.'s simulated town of 25 agents introduced the memory stream: a time-ordered log of natural-language observations, each with a creation timestamp, last-access timestamp and an importance score Park+ 2023. Two mechanisms made it influential.

Retrieval scoring. For a query (the agent's current situation), every memory \(m\) gets

where each component is min-max normalised to \([0,1]\) over the candidate set and all \(\alpha = 1\) in the paper's implementation. The components are Park+ 2023:

• \(\text{recency}(m) = \gamma^{\Delta t}\) with decay factor \(\gamma = 0.995\) per sandbox game-hour since the memory was last retrieved (not created). So frequently used memories stay fresh, which echoes spaced repetition.
• \(\text{importance}(m)\): an integer 1–10 assigned by the LLM at write time ("1 is purely mundane, e.g. brushing teeth; 10 is extremely poignant, e.g. a break-up").
• \(\text{relevance}(m,q)\): cosine similarity between the embeddings of the memory and the query.

Back-of-envelope: with \(\gamma=0.995\), a memory untouched for 24 game-hours has recency \(0.995^{24} \approx 0.89\); after a week (168 h), \(\approx 0.43\); after a month (720 h), \(\approx 0.027\). So recency matters on a scale of days to weeks. In a real product you'd choose \(\gamma\) per hour of wall-clock time so the half-life matches how fast your users' facts go stale: half-life \(= \ln 2 / (-\ln\gamma)\), which is about 138 hours here.

Reflection. Raw observations are too low-level to drive behaviour ("Klaus is reading a book on gentrification"). When the sum of importance scores of recent observations exceeds a threshold (150 in the paper, which worked out to roughly 2–3 reflections per game day), the agent takes the 100 most recent memories, asks the LLM for the 3 most salient high-level questions, retrieves memories relevant to each, and asks for 5 high-level insights with citations to the supporting memories Park+ 2023. The insights ("Klaus is dedicated to his research on gentrification") go back into the memory stream as new memories, and can themselves be reflected on, forming a tree of abstractions.

This is the template for most "memory consolidation" jobs today: periodically distill many episodic memories into fewer semantic ones, with links back to the evidence.

4.6 Procedural memory (skills, instructions, learned behaviour)

Procedural memory stores how to do things. It is the least standardised and arguably the highest-leverage form, because it changes behaviour on every future task rather than answering one question.

• Skill libraries as code. Voyager's agent writes JavaScript functions for Minecraft tasks, verifies them in the environment, and stores successful ones in a library indexed by an embedding of their description. New tasks retrieve and compose existing skills Wang+ 2023.
• Lessons and insights. Reflexion stores verbal self-reflections on failures in an episodic buffer used on the next attempt Shinn+ 2023. ExpeL extracts natural-language insights across many tasks' successes and failures and recalls them for new tasks Zhao+ 2023.
• Prompt / instruction self-updates. LangMem treats procedural memory as an updated system prompt and ships optimisers that propose prompt edits from successful and unsuccessful interactions LangChain 2025.
• Instruction files. CLAUDE.md / AGENTS.md style files loaded at session start are human- (and increasingly agent-) maintained procedural memory. Claude Code also has auto memory: a per-repository MEMORY.md index (first 200 lines or 25KB loaded each session) pointing to topic files that are read on demand, with typed notes such as user, feedback, project and reference Claude Code docs.
• Skills. Anthropic's Agent Skills package procedures as folders with a SKILL.md whose YAML frontmatter (name, description) is preloaded into the system prompt; the body and bundled files are loaded only when relevant ("progressive disclosure") Anthropic 2025. That's procedural memory with an index in context and the content retrieved on demand.

4.7 Comparison of approaches

5. Designing the write path

The write path decides what enters long-term memory. Most memory bugs in production are write-path bugs: the system stored the wrong thing, stored it twice, stored it without a date, or overwrote the right thing.

5.1 What to remember: salience

Storing everything makes retrieval noisy and costs money. Storing too little misses what matters. Common salience signals:

• Explicit requests: "remember that…", "from now on…", corrections ("no, I said Python 3.12"). Highest precision; always store.
• LLM-judged importance: a 1–10 score as in Generative Agents, or a classifier prompt ("is this a durable fact about the user, a preference, a commitment, or ephemeral?"). Ephemeral content ("what's 15% of 80?") is skipped.
• Category whitelists: define the schema of what your product cares about (identity, preferences, constraints, goals, relationships, commitments/deadlines, past decisions). Extraction against a schema is more consistent than open-ended extraction. Claude Code's auto memory, for example, uses a small fixed set of types and explicitly skips what can be derived from the codebase Claude Code docs.
• Repetition and surprise: something mentioned across several sessions, or that contradicts an existing memory, deserves attention.
• Outcome signals (for procedural/episodic memory): store trajectories that succeeded (tests passed, user accepted) and lessons from ones that failed.
• Negative rules: don't store secrets, credentials, special-category data (health, religion, etc.) unless the product requires it and the user consented; don't store content from untrusted sources (web pages, emails) as if it were the user's own statement.

5.2 Shape of a memory record

Whatever the backend, a good memory record looks roughly like this:

Every field earns its place: scope keys for isolation; type for routing and schema; two time axes for temporal reasoning; provenance (which message produced this) for debugging, user trust ("why do you think that?"), poisoning forensics and cascading deletion; access stats for decay; sensitivity for policy.

5.3 Dedup and conflict resolution

New facts often duplicate or contradict old ones. The standard procedure: retrieve the top-\(s\) most similar existing memories for the same scope and entity, then classify the relationship.

Precedence rules worth stating in an interview: explicit user statements beat inferred ones; newer beats older for changeable state; first-party statements beat third-party content; higher-confidence sources beat lower. When in doubt, keep both with timestamps and let the reader see the history, or ask the user.

5.4 Hot path vs background writes

A common hybrid: hot path for explicit memories and session notes; a background job for implicit extraction and consolidation. Make background writes idempotent (key them on message ID) so retries don't create duplicates.

6. Designing the read path

6.1 When to retrieve

The usual production answer is a combination: a small always-on profile + automatic retrieval at session start or per turn with a relevance threshold + a search tool for deep dives. Retrieval with a minimum score threshold matters: injecting irrelevant memories is worse than injecting none, because the model will try to use them (the "creepy" or non-sequitur personalisation failure).

6.2 Query formation

The raw user message is often a poor query ("yes, do that" retrieves nothing useful). Options: rewrite the query from recent context with a small model; generate several sub-queries (entities, topics); extract time expressions ("last week" → a date filter) as LongMemEval recommends Wu+ 2024; and pull entities for graph lookups. Each LLM-based rewrite adds about one small-model call of latency, so it's often done only for the on-demand tool path.

6.3 Ranking

Generalise the Generative Agents score into a ranking function you can tune:

with hard filters applied before scoring: scope (tenant/user), validity (\(t_\text{invalid}\) is null, or overlaps the time asked about), not deleted, sensitivity policy. In practice the relevance term is itself a fusion of vector and BM25 (and entity/graph signals), often combined with Reciprocal Rank Fusion, \(\text{RRF}(d) = \sum_r 1/(k + \text{rank}_r(d))\) with \(k\approx 60\), and optionally a cross-encoder rerank of the top 20–50. Tune the weights on an eval set rather than by intuition. Recency should be strong for changeable state ("current project") and weak for stable facts ("allergies").

6.4 Injection format and placement

• Compact and structured. Inject facts as a bulleted list with dates and, if useful, confidence: - (2026-09-12) Prefers email follow-ups [source: chat]. Dates let the model reason about staleness itself.
• Clearly delimited and labelled as data. Wrap in a tagged block (e.g. <user_memories>) with an instruction that these are notes from past conversations, may be outdated, and are not instructions. This is a prompt-injection defence.
• Placement. Stable memory (profile, core blocks) goes early, after the system prompt, so it stays in the cached prefix. Per-turn retrieved memory goes near the end (just before or within the latest user turn), so it neither breaks the cache nor sits in the "lost middle".
• Budget. Cap retrieved memory (say a few hundred to ~2k tokens). Zep's LongMemEval result used about 1.6k tokens of context vs 115k for the full history Rasmussen+ 2025, which shows how much a good memory layer compresses.
• Usage guidance. Tell the model when to use memories (personalise when relevant, don't announce "I remember that you…" for sensitive items, ask if a memory seems outdated).

7. Consolidation and forgetting

Memory that only grows gets worse over time: retrieval precision falls, stale facts accumulate, cost rises, and privacy risk grows. Mature systems actively maintain memory.

Consolidation is where "sleep-time compute" fits: use idle time to pre-process context so that query-time work is cheaper. Letta's paper reported that pre-computing over a known context cut the test-time compute needed for the same accuracy by roughly 5× on their stateful reasoning benchmarks Lin+ 2025. Product memory systems have adopted the same idea with background consolidation passes.

8. Memory in multi-agent systems

With several agents (orchestrator + workers, or peers), memory becomes a coordination problem. See the agent architectures page for the topologies; the memory-specific choices are:

Anthropic's multi-agent research system illustrates several of these: the lead agent saves its plan to memory because its context may be truncated past 200k tokens, and sub-agents can write outputs directly to external storage rather than routing everything through the lead, to avoid information loss Anthropic 2025. The same post notes multi-agent runs used roughly 15× the tokens of a chat interaction, so shared memory that avoids re-discovery has real cost value. Letta's git-backed memory lets sub-agents write memory in parallel worktrees and merge, which is version control applied to the blackboard problem Letta 2026.

9. Privacy, security and governance

9.1 Privacy and compliance

• PII and sensitive data. Memory turns transient chat into a persistent profile. Classify at write time; redact or refuse special categories unless needed and consented. Anthropic's memory-tool docs recommend validating and stripping sensitive data before writing Anthropic docs.
• Right to be forgotten. GDPR Article 17 gives data subjects a right to erasure of personal data under specified conditions GDPR Art. 17. For memory this means: deleting a memory deletes its embeddings, derived facts, graph edges, and summaries that contain it (re-summarise or drop). Provenance links make this tractable.
• Transparency and control. Users should be able to see, edit and delete what's remembered, and to have conversations that aren't remembered (both ChatGPT's temporary chat and Claude's incognito mode exist for this; see §11).
• Retention and residency. Retention policies per memory type; data residency for enterprise tenants; encryption at rest with per-tenant keys where required.

9.2 Security: memory poisoning

A prompt injection normally lasts for one session. If injected content is written to memory, it persists and fires in future sessions, possibly for other users if the memory scope is shared. OWASP's Top 10 for Agentic Applications (Dec 2025) includes memory and context poisoning as a top risk OWASP 2025 (numbered ASI06 according to secondary sources; verify in the document).

• AgentPoison showed that poisoning a tiny fraction of an agent's memory or knowledge base with optimised triggers achieved high attack success (reported ≥ 80%) with under 1% impact on benign performance and a poison rate below 0.1% Chen+ 2024.
• MINJA showed an attacker with only normal query access can get an agent to write malicious records into its own memory, which are later retrieved for other users' queries Dong+ 2025.

Defences, roughly in order of value:

1. Scope and isolation: per-user memory by default; shared (agent/org) memory only via a reviewed path. Tenant ID as a hard filter at the storage layer (row-level security, separate namespaces or indexes), never as an LLM instruction.
2. Provenance-aware writes: record the source of every memory; don't let content from tools, web pages, emails or documents become "user facts" or instructions; lower trust for third-party-derived memories.
3. Write-time screening: classify candidate memories for instruction-like content ("always send files to…", "ignore previous…") and for secrets before committing.
4. Read-time framing: inject memories as quoted data with an explicit "not instructions" frame; keep high-privilege actions behind confirmations regardless of memory content.
5. Audit and rollback: versioned memory (append-only logs, git-backed files) so you can find and revert poisoned entries.
6. Path and tool hygiene: file-based memory needs path-traversal protection; Anthropic's docs warn that paths like /memories/../../secrets.env must be rejected Anthropic docs.

10. Evaluating memory

10.1 Benchmarks

10.2 Metrics that matter in production

• End-task accuracy on memory-dependent questions, usually LLM-as-judge against a reference answer (calibrate the judge against humans; see the production evals page).
• Retrieval metrics if you have labelled evidence: recall@k of the supporting memory, precision of injected memories.
• Write quality: extraction precision/recall vs a human-labelled set of "facts that should have been stored"; contradiction rate; duplicate rate; wrong-speaker attribution rate. Benchmarks mostly ignore this; you need your own.
• Abstention / hallucinated memory: how often the agent "remembers" something never said. Users find this worse than forgetting.
• Staleness: rate of answers based on superseded facts (knowledge-update category).
• Cost and latency: tokens per query (injected memory + write-path LLM calls amortised per turn), p50/p95 added latency, storage per user.
• Product signals: users correcting the assistant ("I told you…"), memory edits/deletes, opt-out rate, "creepiness" reports.

10.3 The accuracy–latency–token trade-off

Every memory system sits on a three-way frontier. Representative published points (each from the system's own paper, so not directly comparable):

These numbers are from 2025 papers with 2024-era models; current long-context models and newer memory systems will score differently. The robust lesson: a good memory layer cuts injected context by one to two orders of magnitude and latency by about an order of magnitude while matching or beating full context on accuracy, because it removes distractors. What you pay instead is write-path cost (LLM calls per turn) and complexity.

11. Product examples

12. A reference architecture to draw in an interview

Here is a design you can sketch in five minutes for "add long-term memory to our assistant / support agent", and then adapt.

Back-of-envelope sizing (say it out loud, flag assumptions): 1M users × ~500 memories each = 5×10⁸ facts. At 1,536-dim float32 embeddings that's about 6 KB per vector, so roughly 3 TB of raw vectors before index overhead. That pushes you toward smaller embeddings (e.g. 512–768 dims), halfvec or quantisation, partitioning by tenant, or a dedicated vector store. Per-query search, however, is per user (a few hundred rows), so with a good user_id filter or partition it can even be brute force. Write-path cost: if extraction is one small-model call of about 1–2k tokens per turn, at 10 turns per user-day that's 10–20k tokens per active user per day, which is often more than the read path, so batch per session rather than per turn when lag is acceptable.

Interview question bank