Token Budgeting, Compression & Selective Loading

Fixing the Coding Agent

Everything so far in this phase diagnosed the problem: an undifferentiated context window (context architecture) with no mechanism for moving facts in and out of it deliberately (memory systems). This guide is the actual fix the coding agent's team shipped — and it's less about clever prompting than about explicit budgets and eviction rules, applied the same way you'd manage any other finite, shared resource.

Give Every Section of Context a Budget

The fix starts by treating the context window as a fixed resource split across the categories from the first guide in this phase — system instructions, tool definitions, conversation history, retrieved content — instead of letting history grow unbounded and consume whatever's left:

That last row is the direct fix for the stale-tool-output problem from the first guide: a file's contents, once read, don't need to stay in context verbatim for the next 30 turns — only the current state matters, and re-reading a file is cheap compared to carrying every historical version forward.

Three Eviction Strategies

Once there's a budget, something has to enforce it when history exceeds it. Three approaches, usually combined rather than used alone:

1. Sliding window — keep only the most recent N turns verbatim, drop the rest. Simplest to implement, cheapest to reason about, and lossy in an unstructured way: nothing decides which old information mattered before it gets dropped.
2. Summarization-based compaction — periodically collapse older turns into a compact summary that preserves the decisions and facts that still matter, discarding the verbose back-and-forth that produced them. This is what actually solved the coding agent's contradicting-instruction problem: turn 15's superseded instruction doesn't need to survive verbatim, but "the user wants X, not Y, as of turn 22" does — and a summarization pass can encode exactly that resolution instead of leaving both versions sitting in context unlabeled.
3. Importance-based retention — explicitly tag some content (a pinned instruction, a critical decision) as exempt from eviction regardless of age, while everything else ages out normally. This is what long-term memory (previous guide) effectively is: a permanent exemption from the eviction that governs everything else.

Compression: Fewer Tokens, Same Meaning

Eviction decides what stays. Compression reduces how many tokens what stays actually costs. LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models introduced a practical approach: use a small, cheap model to estimate which tokens in a prompt carry the least information, and strip them out before sending the (much shorter) result to the expensive model — often cutting prompt length substantially with limited quality loss, because natural language carries real redundancy that a model doesn't strictly need to preserve meaning. For a verbose tool output — a full file listing, a raw API response — this is frequently a better fit than either keeping it verbatim or dropping it entirely.

Selective Loading: Don't Load It Until It's Needed

The last piece closes the loop with the memory systems guide: instead of deciding upfront what a session might need and loading all of it, selective loading defers loading until a specific turn actually calls for it — the same just-in-time principle semantic memory applies to facts, applied here to any context content. A coding agent doesn't need the entire codebase's file tree resident in context for the whole session; it needs the specific files relevant to the current step, loaded when that step starts and evicted once it's done. This is the same instinct behind retrieval-augmented generation in Phase 4 — retrieve what's relevant to now, not everything that might theoretically matter.

---

Back to the Coding Agent

The fix the team shipped combines all three ideas from this phase: context assembly with clear sections and budgets (first guide), long-term memory for facts that should persist across sessions and structured semantic memory for facts worth retrieving on demand (second guide), and — this guide — summarization-based eviction of stale tool outputs plus pinning the current task's instructions as exempt from that eviction. The agent stays coherent past 150 turns instead of degrading around 40, and average token spend per session drops roughly 60%, because it's no longer paying to re-send file contents and resolved decisions it never needed to carry forward in full.

None of this replaces good retrieval or good prompting — it's the layer that makes both of those actually reliable once a session runs long enough for "just include everything" to stop being an option. Phase 4 picks up the retrieval half of this story directly: how to decide what to bring into context from an external knowledge base in the first place, which is the same selective-loading instinct from this guide, applied to documents instead of an agent's own history.