Context Architecture & Assembly
What belongs in a context window — instructions, tool definitions, history, retrieved documents — and why the order you assemble them in changes model behavior.
Bizarre Edits at Turn 40
A coding agent handles the first thirty or so turns of a refactoring task fine. Somewhere past turn 40, it starts doing strange things — re-reading files it already edited, reintroducing a bug it fixed ten turns earlier, following an instruction the user explicitly countermanded halfway through the session. Nobody changed the model. Nobody changed the prompt template. What changed is everything around the prompt: by turn 40, the context window is a scroll of every tool call, every file read, every intermediate thought the agent has ever had in this session, with no curation applied to any of it.
This is context rot, and it isn't a mysterious degradation — it's the direct, mechanical result of treating the context window as an append-only log instead of something deliberately assembled. Phase 1 covered why a bigger window doesn't guarantee even attention across everything in it (Lost in the Middle found models recall the start and end of a long context far better than the middle). This guide is about the discipline that actually prevents the problem: deciding, on purpose, what goes into context, and in what order — rather than defaulting to "everything, always."
What Actually Belongs in Context
Every request to a model assembles several distinct kinds of content into one token stream, and each behaves differently:
| Component | What it is | How often it changes |
|---|---|---|
| System instructions | Persona, ground rules, current task | Rarely — should stay stable across a session |
| Tool definitions | Schemas for available actions | Static per session, sometimes per turn |
| Conversation history | Prior user/assistant turns | Grows every turn — the main growth driver |
| Retrieved content | Documents, tool outputs, search results | Volatile — often only relevant for 1–2 turns |
Treating these as one undifferentiated blob is exactly what happened to the coding agent: tool outputs from turn 12 (the contents of a file, since edited twice more) were still sitting in context at turn 40, competing for the model's attention with the actual current state of the file and the actual current instruction. The model wasn't malfunctioning — it was doing next-token prediction over a context that genuinely contained contradictory information, because nothing had ever removed the stale version.
Order Isn't Neutral
Given the recency and primacy effects covered in Phase 1, where something sits in the assembled context measurably affects how much weight the model gives it — not just whether it's present at all. A few practical consequences that follow directly from that:
- Pin stable instructions where they won't get buried. System-level ground rules that must hold for the entire session are the content most vulnerable to being drowned out once history grows long enough to push them far from either end of the window.
- Put the most decision-relevant content close to where the model has to act. If a tool result is the thing the next response actually depends on, its position relative to the final user turn matters, not just its presence somewhere in the window.
- Don't let stale and current information coexist un-flagged. The countermanded instruction in the coding agent's history wasn't wrong when it was said — turn 15's instruction was correct until turn 22 overrode it. The context had no signal marking turn 15's version as superseded, so the model had no structural reason to prefer turn 22's version over turn 15's.
Anthropic's Effective Context Engineering for AI Agents and LangChain's The Rise of Context Engineering both make the same underlying argument from different angles: as agent sessions get longer, what you deliberately choose to include — and exclude — matters more than prompt wording ever did for single-turn tasks.
Check yourself
The coding agent reintroduces a bug it already fixed, because an early tool output showing the buggy file version is still present in context alongside the later, corrected version. What's the most accurate description of the failure?
What's Next
Deciding what belongs in a single request's context is one half of the discipline. The other half is what persists across turns and sessions at all — which facts a system should remember on purpose, versus which should simply be re-derived or dropped. That's memory, and it's a different mechanism from the context assembly covered here.
Frequently asked questions
Isn't 'just include everything, the model is smart enough to figure out what matters' a reasonable default?
It's the default that caused the coding agent's failure. Models weight recent and boundary content more reliably than content buried in the middle of a long context, and contradictory information left uncurated has no structural signal telling the model which version to trust. Scale and 'smart enough' don't override that — it's an architectural property of how attention behaves over long contexts, covered in Phase 1.
Does context architecture replace RAG?
No — they solve different problems. RAG (Phase 4) decides which external documents to retrieve into context in the first place. Context architecture decides what to do with everything already inside the context window: what to keep, what to evict, and how to order it. A well-architected context still needs good retrieval feeding into it.
How often should system instructions change mid-session?
As rarely as possible. Instructions that shift mid-session are exactly the kind of content that creates the 'contradicting instructions' failure mode — if the task genuinely changes, it's often clearer to explicitly supersede the old instruction in the context (mark it as replaced) rather than silently adding a new one on top of the old.