Memory Systems: Short-Term, Long-Term & Semantic
Why a session's conversation buffer, a user's persisted preferences, and a searchable store of facts are three different memory mechanisms, not one.
Two Different Kinds of Forgetting
The coding agent from the last guide has a second problem, quieter than the context-rot one. Every session starts from zero: it re-asks which linter the project uses, re-discovers that this team writes tests before implementation, re-learns that this particular user always wants explanations before code changes. None of that is session-specific working data — it's the same handful of facts, true across every session with this user, and the agent has no mechanism for carrying it forward. Context architecture (the last guide) governs what's assembled within one request. This guide is about a different question: what should persist across requests, and even across sessions, at all.
"Memory" in an LLM system isn't one mechanism — it's three, and conflating them is where most homegrown implementations get complicated for no reason.
Short-Term Memory: The Conversation Buffer
This is the memory everyone builds first, often without thinking of it as "memory" at all: the running list of messages resent on every call within a session, because — as covered in Phase 1 — the model itself remembers nothing between requests. It's ephemeral by design. Close the session, and short-term memory is gone unless something explicitly persists it. Its main engineering concern isn't sophistication, it's the token-budgeting problem the next guide covers: an unbounded buffer is exactly what turned into context rot for the coding agent.
Long-Term Memory: Facts That Outlive a Session
Long-term memory is deliberately persisted state that survives past the session that created it — a user's linter preference, a project's testing convention, a standing instruction like "always explain before changing code." The distinguishing property isn't how it's stored (a database row, a key-value store, a vector index can all work) — it's that something in the system decided this fact was worth keeping on purpose, separately from the transient conversation log, and retrieves it deliberately at the start of a new session rather than rediscovering it from scratch.
MemGPT: Towards LLMs as Operating Systems (Packer et al., 2023) frames this well as an OS-style paging problem: the context window is scarce, fast "RAM," and long-term storage is cheap, slow "disk" — a well-designed agent moves facts between the two deliberately, the same way an OS pages memory in and out, rather than trying to keep everything permanently resident in the expensive tier. That framing carries directly into the next guide's compression and eviction strategies.
Semantic Memory: Retrieval Instead of Injection
Long-term memory as described above still implies loading facts wholesale at session start — fine for a handful of stable preferences, unworkable once the volume of things worth remembering grows past what's cheap to inject into every request. Semantic memory solves that by storing facts as embeddings (Phase 1's latent space, applied to memory instead of documents) and retrieving only the handful relevant to the current turn, the same retrieval mechanism Phase 4 covers for RAG, applied to a memory store instead of a document corpus. Mem0 and LangGraph's memory concepts are two current implementations of this pattern — extracting durable facts from conversations, indexing them, and recalling only what's relevant on each new turn instead of replaying full history.
The practical distinction that actually matters day to day: long-term memory is what you always load; semantic memory is what you look up. A user's name is worth always loading — it's small and universally relevant. A user's opinion about a database migration from three months ago is worth indexing and retrieving only when a related topic comes up — loading it into every unrelated conversation would just be more context-rot fuel.
Check yourself
A coding agent needs to recall that this user always wants unit tests written before implementation code (true in every session) and also needs to recall a specific debugging decision from a session two months ago, only when a related bug resurfaces. Which memory mechanism fits each?
What's Next
Both long-term and semantic memory reduce how much needs to live in the conversation buffer — but the buffer itself still grows every turn, and MemGPT's RAM/disk framing only works if something actually decides what moves where and when. That's token budgeting, compression, and selective loading: the mechanism that turns "we know what memory tiers exist" into "the coding agent stays coherent past 150 turns."
Frequently asked questions
Do I need a vector database to implement memory, or is a regular database enough?
For long-term memory — a small, fixed set of facts always loaded at session start — a regular database or even a config row is often enough; there's no retrieval problem to solve if you're always loading everything. Semantic memory specifically needs similarity search over a growing, unpredictable set of facts, which is what a vector store is for.
How does an agent decide what's worth writing to long-term memory in the first place?
Usually either an explicit user action (settings, stated preferences) or a periodic extraction step where the model itself is asked to identify durable facts from a completed session — 'what should be remembered from this conversation for next time?' Neither approach is perfect; extraction quality is an active area of tooling development, which is part of why Mem0 and similar libraries exist rather than everyone hand-rolling it.
Is short-term memory ever worth persisting?
Sometimes — a user resuming an interrupted session benefits from their in-progress conversation surviving a page refresh, which is really short-term memory backed by storage rather than a new memory tier. The distinction that matters is intent: is this the current working state of one session, or a fact meant to outlive it? The former is still short-term memory even if it's technically saved somewhere.