System Design for LLM Applications: The Architecture Interviews Don't Teach Yet

The classic system design interview asks you to design Twitter's feed, a URL shortener, or a ride-hailing dispatcher. You reason about read/write ratios, sharding keys, caching layers, consistency models. The tools are well-worn and the failure modes are understood.

Now design a support assistant that answers from your company's documentation. Most of the classic reasoning still applies — it's a web service with a database and a cache. But the dependency at the center behaves unlike anything else in your architecture:

• It's non-deterministic. The same input can produce different output.
• It's expensive per call — often 10,000× the cost of a database read.
• It's slow — seconds, not milliseconds, and variable.
• It fails silently. A wrong answer looks exactly like a right answer.
• Its capacity is rate-limited by a vendor, not by hardware you can add.

Every architectural decision in an LLM system flows from those five properties. Here's how the pieces fit.

The Reference Architecture

Before the details, the shape most production LLM systems converge on:

The interesting thing about this diagram is how little of it is "the model." The model call is one box. The other seven are where your engineering goes — and where systems fail in production.

1. Token Budgeting Is Your Capacity Planning

In a traditional service you plan capacity in requests per second and memory per instance. In an LLM service, the equivalent unit is the token budget per request, and it is simultaneously your cost model, your latency model, and your quality model.

Treat the context window as a fixed budget you allocate deliberately, not a bucket you fill until it's full:

Two failure modes I've watched teams hit repeatedly. The first is unbounded history: a chat session that appends every turn works fine in testing and quietly triples per-request cost by turn thirty. The fix is a compaction policy decided at design time — sliding window, running summary, or summary-plus-recent-turns.

The second is assuming more context is better. It isn't. Retrieval quality degrades measurably when you stuff marginally-relevant chunks in alongside the good ones, because the model now has to find the signal. Fewer, better chunks beat more chunks — which is the precision/recall trade-off showing up as an architecture decision.

2. Semantic Caching: The Highest-Leverage Component

In a normal service, caching is an optimization. In an LLM service it's structural, because the thing you're avoiding costs a thousand times more than the cache lookup.

But exact-match caching barely helps — natural language means users phrase the same question fifty ways. Semantic caching keys on meaning: embed the query, search for a previous query above a similarity threshold, and return the stored answer on a hit.

query → embed → vector search over cached queries
      → similarity > threshold ? return cached answer : call model, then store

Three things decide whether this helps or hurts:

The threshold is a precision/recall dial with real consequences. Set it loose (0.80) and you'll serve the answer to a similar but different question — the worst possible bug, because it's confidently wrong and invisible in your metrics. Set it tight (0.97) and hit rates collapse. Start conservative, measure false-hit rate by sampling, and loosen only with evidence.

Cache scope is a security boundary. If answers depend on who's asking — tenant, role, permissions — the cache key must include that identity. A global semantic cache over multi-tenant data is a data leak with a TTL. This is the same problem as access control in production RAG, and it's the single most common design flaw I see in LLM architectures.

Invalidation is harder than in a normal cache. When the underlying document changes, which cached answers are stale? You usually can't tell precisely, so the practical approach is conservative TTLs plus bulk invalidation keyed to source-document versions.

For assistants over a stable knowledge base, hit rates of 30–40% are achievable — which is a 30–40% cut in both your bill and your p50 latency. Few other single components do that.

3. Model Routing: Stop Sending Everything to the Frontier Model

Not every request needs your most capable model. "Reset my password" and "reconcile these three contradictory policy documents" arrive through the same endpoint and should not cost the same.

A router classifies the request and dispatches accordingly:

• Tier 0 — no model. Deterministic intents matched by rules or classifier. Cheapest possible path; use it.
• Tier 1 — small/local model. Classification, extraction, short factual answers over retrieved text.
• Tier 2 — frontier model. Multi-step reasoning, synthesis, anything customer-visible and high-stakes.

The classifier itself must be cheap — a small model or an embedding-based classifier, never a frontier call, or you've paid the price you were avoiding. Two design rules matter more than the tiering itself:

1. Escalation must be possible. Tier 1 needs a way to say "I'm not confident," and the router needs to retry at Tier 2. Without it, routing trades cost for quality silently.
2. Route on evaluated evidence, not vibes. Build the eval set first (§5), then measure which request classes the small model actually handles. Routing decisions made by intuition are how quality regressions ship.

4. Failure Design: Fallback Chains and the Silent-Failure Problem

Your model provider will rate-limit you, degrade, or go down. Unlike a database outage, this happens on someone else's schedule with no warning. Design for it explicitly:

Two hard-won rules. Retries need jitter and a budget — an LLM timeout is often 30+ seconds, so three naive retries means a 90-second request holding a connection while your thread pool drains. Cap total time, not just attempt count. And a multi-provider fallback is only real if you've tested it — if the secondary provider's prompt format, tool-calling schema, or output shape differs, your fallback path fails on its first real invocation. Exercise it in production regularly, the way you'd exercise a DR plan.

Then there's the failure mode with no analogue in traditional systems: the model returns a fluent, well-formatted, completely wrong answer with a 200 status code. No exception, no error rate, no alert. Your dashboards stay green while users get bad information.

The mitigations are structural, not reactive:

• Structured outputs with schema validation. Make the model return JSON against a schema and reject non-conforming responses. This converts a class of silent failures into loud ones.
• Grounding checks. For RAG, verify the answer's claims are supported by the retrieved context. A cheap model scoring "is this answer supported by this context" catches a meaningful fraction of hallucinations.
• Confidence-aware refusal. A system that says "I don't have information on that" is more valuable than one that always answers. Design the refusal path deliberately; don't leave it to the prompt.

5. Evaluation Is Part of the Architecture, Not the QA Plan

This is the piece most teams bolt on last and regret. In a deterministic system, tests pass or fail. In an LLM system, a prompt tweak improves three cases and breaks two others, and without an eval pipeline you will not know which.

What the pipeline needs, structurally:

• A golden dataset — 50–200 real inputs with known-good outputs, drawn from production traffic rather than invented. Start here; it's the highest-value artifact in the whole system.
• Deterministic checks first — schema validity, citation presence, latency, cost, refusal rate. Cheap, fast, and they catch more than people expect.
• LLM-as-judge for subjective quality — with a rubric, and periodically calibrated against human ratings so you know the judge is trustworthy.
• Regression gates in CI — prompt and model changes run the eval set before merge, the same way code changes run unit tests.
• Online sampling — score a percentage of live traffic, because production drifts from your golden set.

Treat the prompt as source code: versioned, reviewed, and tested. Prompts edited directly in a production config panel are the LLM equivalent of SSH-ing into a box to patch a binary.

6. Latency Architecture

LLM latency is dominated by generation — time to produce N output tokens — which means the usual optimizations barely move it. What does:

• Stream everything user-facing. Time-to-first-token is the number users perceive. A 4-second response that starts rendering at 400ms feels dramatically faster than a 2-second response that appears all at once.
• Parallelize context assembly. Retrieval, memory lookup, and user-profile fetch are independent; run them concurrently while the model call waits.
• Move work off the critical path. Logging, eval scoring, and analytics belong on an async queue, never in the request.
• Cap output length deliberately. Output tokens dominate generation time. "Answer in under 150 words" is a latency optimization with a quality trade-off, and it should be a conscious one.

What This Means for Interviews

If you're preparing for system design interviews, the LLM-flavored questions ("design an AI customer support system," "design a code review assistant") are increasingly common and most candidates answer them as if the model were a normal service dependency. The signal interviewers are looking for:

• You budget tokens the way you'd budget IOPS.
• You reach for semantic caching and name its failure mode.
• You route by request class instead of sending everything to the biggest model.
• You design for silent wrongness, not just for 5xx.
• You bring up evaluation unprompted.

That last one separates people who've shipped an LLM feature from people who've built a demo. Demos don't need evals. Systems do.

Go deeper: the System Design roadmap covers the fundamentals this post builds on, and the Gen-AI Systems phase works through LLM gateways and routing, prompt and semantic caching, guardrails, and LLM observability and evaluation in detail.