Cost Optimization & Model Routing
The bill is growing faster than usage justifies — prompt caching, semantic caching, and routing simple requests away from the expensive model.
The Bill Is Growing Faster Than Usage
Rolling out from a 50-customer beta to the full customer base, someone notices the API bill isn't scaling linearly with request volume — it's scaling faster. Two habits are responsible, and neither was a problem at beta scale: every single request re-sends the full system prompt and every tool definition from scratch, paying full price to reprocess text that hasn't changed since the last request, and every request — a two-second "what's my order status" lookup and a genuinely complex multi-step dispute alike — runs on the same top-tier, most expensive model regardless of how much reasoning it actually needed.
Prompt Caching: Don't Re-Pay for What Didn't Change
Phase 3 established that a request's context has a stable part (system instructions, tool definitions) and a volatile part (the current conversation turn). Prompt caching exploits exactly that split: the model provider caches the processed representation of a stable prefix — the system prompt and tool definitions that are identical across many requests to the same agent — so subsequent requests only pay full processing cost for the new, volatile part, not the whole prompt from scratch every time.
For an agent with a long, stable system prompt and a dozen tool definitions — exactly the refund worker's profile — this is often the single highest-leverage cost fix available, precisely because that stable prefix was being fully reprocessed and billed on every single call before caching was applied. Anthropic's prompt caching documentation covers the mechanics of how a cached prefix is identified and reused.
Semantic Caching: Reuse Across Similar, Not Just Identical, Queries
Prompt caching reuses a stable prefix. Semantic caching goes further, reusing an entire previous response when a new query is similar enough in meaning to one already answered — using the same embedding-similarity mechanism from Phase 1 and Phase 4, applied to matching incoming queries against a cache of previously answered ones instead of matching queries against documents. "What's your refund policy" and "how do refunds work" are different strings but the same underlying question; a semantic cache recognizes that and returns the already-computed answer instead of running the full pipeline again. The tradeoff is real: semantic similarity is approximate, so an overly aggressive similarity threshold risks returning a cached answer to a question that was subtly different in a way that mattered — this needs the same measured, not guessed, tuning as the ANN speed/recall tradeoff from Phase 5.
Model Routing: Not Every Question Needs the Frontier Model
The second habit — one model tier for every request — is a separate, often larger cost driver. Model routing sends a request to a smaller, cheaper model for simple, well-understood tasks and reserves the expensive frontier model for requests that genuinely need its reasoning capability, decided by a fast, cheap classification step — the same routing pattern as Phase 7's orchestrator, now routing to a model tier instead of a specialized worker agent:
RouteLLM (Ong et al., 2024) studied this pattern directly, training routers that predict which queries a cheaper model can handle at comparable quality to a frontier model — the research finding worth internalizing is that a meaningful share of real-world traffic doesn't need the most expensive model at all, and a well-tuned router captures most of the quality of always using the frontier model at a fraction of the cost, rather than forcing an all-or-nothing choice between cost and quality.
Measure before optimizing. Prompt caching is close to a free win once a system's prefix is genuinely stable; model routing requires the evaluation discipline from the last guide to confirm the cheaper model's outputs on routed traffic actually hold up — a routing change that saves money but quietly degrades quality on the wrong slice of traffic isn't a win, it's a regression wearing a lower invoice.
Check yourself
Why does prompt caching provide a large cost benefit specifically for the refund agent's system prompt and tool definitions, but not for the customer's current message?
What's Next
Cost, evaluation, and observability now all have measurement in place. The last piece is making sure the system that's cheaper and faster to run is also safe to expose to everyone — which is where the pre-launch red-team exercise comes in.
Frequently asked questions
Does semantic caching risk returning stale answers if the underlying policy changes?
Yes, and this is the same freshness problem from Phase 4 applied to a response cache instead of a document index — a cached answer needs an expiration or invalidation strategy tied to when the underlying source content changes, not just a similarity threshold. A cache that never expires will confidently serve an outdated policy exactly the way a stale RAG index would.
How is model routing different from the multi-agent orchestrator from Phase 7?
They're the same underlying pattern — classify, then route — applied at different layers. The Phase 7 orchestrator routes a request to a specialized *worker agent* based on domain (refund vs. shipping vs. account). Model routing here routes a request to a *model tier* based on complexity, and the two can operate independently and simultaneously: a request can be routed to the refund worker, which then further routes its own reasoning step to a cheap or expensive model depending on the specific sub-task.
Is a smaller, cheaper model always worse quality than a frontier model?
Not universally — for well-scoped, low-ambiguity tasks (classification, simple lookups, straightforward extraction), a smaller model can perform close to indistinguishably from a frontier model, which is exactly the traffic RouteLLM-style routers are built to identify. The gap widens specifically on tasks requiring deeper multi-step reasoning, which is why routing — sending the right task to the right tier — outperforms picking one model size for everything.