In This Issue
Why conversation history, context, retrieval, and memory are different system concerns.
A practical test for deciding whether state should persist between turns.
Which state should survive, which should be reconstructed, and which should be retrieved fresh.
A memory contract to define before choosing a database, framework, or agent runtime.
The brief
The easiest way to give an AI system memory is to keep feeding it what happened before.
That works until the history becomes large, stale, contradictory, expensive, or irrelevant to the decision the system is making now.
The harder problem is deciding what deserves to survive.
A useful AI application does not need to remember everything. It needs enough state to continue the work without rediscovering decisions, constraints, and commitments that are still valid.
That makes memory a requirements problem before it becomes a storage problem.
The question is not, “Where should we store the conversation?”
It is, “What information from this interaction should be allowed to influence a future one?”
Context, retrieval, and memory do different jobs
Context is what the model can see for the current inference. It can include the current request, recent messages, retrieved documents, tool results, instructions, structured state, and selected memories.
Retrieval is how the application finds information that is not already in that working context.
Memory is state that survives an interaction so it can influence later work.
The boundaries matter because storing something does not mean it belongs in every future prompt. OpenAI’s Agents SDK can persist session history while still allowing developers to filter what is sent to the model or compact longer sessions. LangGraph similarly distinguishes thread-scoped short-term state from long-term memory shared across conversations, and recommends trimming or summarizing histories when needed.
Persistence and prompt assembly should be separate decisions.
The memory test
I would put every candidate memory through five questions before allowing it to survive the turn.
Question | What it tells you |
|---|---|
Will this change a future decision? | If losing it would make the system repeat work, violate a constraint, or choose differently, it may be worth carrying. |
Is it likely to remain true? | Stable preferences and explicit decisions are stronger candidates than temporary observations. |
Is this system the source of truth? | If the authoritative answer lives elsewhere, retrieve it again instead of memorizing a copy. |
What is the correct scope and lifetime? | State may belong to one task, conversation, project, user, or organization. |
How will it be corrected or forgotten? | If the update and deletion path is undefined, the memory design is incomplete. |
This is requirements work. The database comes later.
What usually deserves to survive
The strongest candidates represent continuity rather than raw history.
A current task may need its objective, selected entity, pending approval, and resume point. A project may need durable constraints such as a chosen technology or an agreed output format. A user-facing application may remember an explicitly provided preference when that preference is useful across future sessions.
The common property is not that these facts appeared in the conversation. It is that forgetting them would change how the system should behave later.
The state must have a future job.
What should be retrieved again
Live business facts are different.
Suppose a support assistant learns that an order is delayed. That delay matters now, but the order system should remain authoritative. When the customer returns three days later, the assistant should retrieve the current status rather than trust a memory saying the shipment is delayed.
The same rule applies to balances, inventory, permissions, prices, repository contents, policy text, and other facts that can change outside the conversation.
The application can remember the identifier and why the item matters. It should usually re-read the truth.
Memory preserves continuity. Retrieval preserves freshness.
Do not turn execution residue into working memory
Agent systems produce a lot of state that is useful for debugging but poor input for future reasoning.
Tool responses, intermediate candidates, retry details, old search results, verbose traces, and failed plans may need to be logged for observability or audit. They do not automatically belong in the model’s next context.
OpenAI’s session guidance reflects this separation. A session can store history while the application still limits or filters what is retrieved for the next run.
Give memory an explicit shape
Free-form summaries are useful, but they should not be the only representation of important state.
For work that spans turns, I would prefer a small structured record that says what the system is trying to do, what has been decided, what remains unresolved, where authoritative data lives, and when the state should expire or be reviewed.
A support workflow might carry a case ID, the user’s stated objective, the last confirmed decision, and a pending approval. It should not carry a cached account balance or every tool response.
Structured state also makes change easier. A new decision can replace an old one. A completed task can close its task state. A temporary assumption can expire instead of quietly becoming permanent history.
Memory also needs a read policy
Writing memory is only half of the system.
At the next turn, the application still needs to decide which stored state is relevant enough to return.
Short task state may be loaded automatically because the next step cannot continue without it. Long-term user or project memory is usually better selected by scope and relevance. External facts should be retrieved from their source when the current decision requires them.
This is why adding a vector database is not, by itself, a memory architecture.
The architecture is the set of rules for what can be written, where it is scoped, how long it lives, what can update it, and when it comes back into context.
The move
Before adding persistent memory to an AI workflow, write a memory contract for one real use case.
Define | Example decision |
|---|---|
State that must survive | Current objective, confirmed constraints, unresolved approval, resume point. |
State that should expire | Temporary assumptions, transient tool output, one-off working notes. |
State that must be retrieved fresh | Current customer record, prices, permissions, policies, repository state. |
Scope | Task, conversation, project, user, or organization. |
Write authority | User, deterministic application logic, approved agent action, or background extraction. |
Update and deletion path | Replace, supersede, expire, close with task, or delete on request. |
Read rule | Always load, load for this task, retrieve by relevance, or fetch from source of truth. |
Then test an awkward follow-up: an external fact changes, a preference is corrected, or an earlier decision is reversed.
If the system cannot tell which state should win, the memory requirement is incomplete.
Worth reading
OpenAI’s Agents SDK documentation on sessions is useful because it shows the implementation boundary clearly: history can persist across runs without forcing the application to send all stored history to the model every time. It also covers filtering, limiting, compaction, and multiple storage backends. OpenAI Agents SDK: Sessions
Takeaways
If you are deciding... | Use this rule |
|---|---|
Whether to persist something | Keep it only if losing it would change a future decision or force meaningful work to be repeated. |
Whether to memorize an external fact | Prefer a pointer and retrieve the current value from the authoritative system. |
Whether stored history belongs in the next prompt | Select what is relevant. Persistence does not imply inclusion. |
How long memory should live | Match lifetime to scope. Task state should not quietly become permanent user memory. |
Whether the design is complete | Make correction, supersession, expiration, and deletion explicit requirements. |
The goal is not a system with more memory. It is a system that carries the minimum state required to continue correctly.
If this helped you, leave a comment or your reaction. I’d like to hear where you landed.
INVENEW exists to help tech builders, operators, founders, and leaders turn AI from experiments into working systems.
In partnership with AWS
Sponsored link – Webinar: Redefining application and API security in the age of AI. See how to embed security early, align with compliance expectations, and discover AWS Partner solutions in AWS Marketplace.
Note: Third-party company and product names belong to their respective owners and are used for identification and illustrative reference only.