Small Models and Inference Cost Engineering: When a 7B Beats a Frontier API

The default architecture for an LLM feature is an API call to a frontier model, and for good reason: it's one line of code, it works on the first try, and the model is better than anything you could run yourself. I'd make that choice again for almost every prototype.

The problem shows up at scale. A feature that costs a fraction of a cent per call is free at 1,000 calls a day and a budget line item at 10 million. At that point somebody asks whether a smaller model could do the job — and the honest answer, for a surprising share of production workloads, is yes.

This post is about how to tell which workloads those are, and what self-hosting actually costs once you count the parts people leave out.

First: Most Workloads Don't Need a Frontier Model

Frontier models are optimized for the hardest thing you might ask. Production traffic is mostly not the hardest thing you might ask.

Look at what an LLM feature actually does over a week and you'll typically find a long tail of easy, repetitive work:

• Classification — intent, sentiment, routing category, priority. A fine-tuned small model frequently beats a frontier model here, because the task is narrow and the training data is yours.
• Extraction — pulling structured fields out of text against a known schema.
• Short-form generation — titles, summaries, subject lines, tags.
• Reformatting — text in one shape to text in another.
• Embedding — never a frontier-model job to begin with, though plenty of teams accidentally route it like one.

And the genuinely hard work that earns a frontier call:

• Multi-step reasoning over conflicting information
• Long-horizon agentic planning
• Nuanced writing where quality is the product
• Open-ended synthesis across many documents
• Anything where being wrong is expensive

The architectural move is to stop treating these as one workload. Model routing sends the easy tiers to cheap models and reserves the frontier model for calls that need it. Most teams that measure find well over half their traffic sits in the easy tiers.

The Honest Cost Math

Here's where most "self-hosting saves money" analyses go wrong: they compare API token prices against GPU rental and stop there.

The real comparison:

The decisive line is "scales to zero." An API bill tracks usage; a GPU bills 24/7 whether or not anyone uses your feature. Self-hosting is a fixed cost, so it only wins above a utilization threshold — and the threshold is higher than people expect, because idle GPU time is pure waste.

The rough shape, which you should recompute with your own numbers rather than trusting mine:

• Low, bursty volume → API. Not close.
• Moderate steady volume → hosted small model / open-weights inference provider. This is the sweet spot most teams miss entirely: you get 10–100× lower token costs, still scale to zero, and write no infrastructure.
• High, steady, predictable volume → self-hosting starts to win, if you actually keep the GPU busy.
• Any volume where data cannot leave your network → self-hosting, and the cost question is moot.

That middle row deserves emphasis. Teams tend to frame the decision as "frontier API vs. run it ourselves," skipping the option that's right for most of them — a small open-weights model served by someone else, at a fraction of frontier pricing and none of the operational burden.

Quantization: The Highest-Leverage Optimization

If you do run models yourself, quantization is the first lever and the one with the best return. It stores weights at lower numeric precision — 16-bit down to 8-bit or 4-bit — shrinking memory and increasing throughput.

The practical trade-off, in round terms:

The consequence that matters: quantization changes which hardware you need. A model that requires a large data-center GPU at FP16 may fit on a much cheaper card at INT8, or on a consumer GPU at INT4. That's often the difference between "we can't afford to self-host" and "we can."

Two cautions from experience. Quality loss is task-dependent — INT4 that's fine for classification can visibly degrade multi-step reasoning or code generation. Always evaluate the quantized model on your eval set, not on published benchmark deltas. And quantization interacts with long context: the KV cache, which grows with context length, can end up dominating memory even after the weights shrink, so measure at your real context sizes.

The Serving Stack

Running a model in production is not model.generate() in a loop. Naive serving wastes most of your GPU, which erases the savings you self-hosted for.

vLLM is the default for production serving, and the reason is continuous batching: instead of waiting to assemble a fixed batch, it schedules incoming requests into the batch as slots free up, keeping the GPU saturated under variable load. Paired with PagedAttention — which manages the KV cache in pages rather than contiguous blocks, cutting memory fragmentation — it delivers many times the throughput of naive serving on identical hardware. If you're self-hosting for cost reasons, this is not optional.

Ollama is the local-development and single-user tool. Trivial setup, sensible defaults, ideal for evaluating candidate models on a laptop, prototyping offline, or a desktop app that runs models on the user's machine. It's not built for concurrent production traffic; don't put it behind a load balancer and call it a serving tier.

Practical throughput levers beyond the framework choice:

• Continuous batching — the single biggest one, above.
• Prefix caching — if every request shares a long system prompt, cache its KV representation instead of recomputing it per request. For RAG systems with fixed instructions, this is a large, easy win.
• Speculative decoding — a small draft model proposes tokens, the big model verifies in parallel. Meaningful latency reduction for the same output.
• Right-sized context limits — an 8K cap instead of 128K when 8K is all you need frees KV-cache memory that becomes throughput.

Distillation and Fine-Tuning: Buying Quality Back

Routing sends easy traffic to a small model as-is. Fine-tuning makes the small model genuinely good at your task, which widens the set of traffic you can route away from the frontier.

The pattern that works, and it's pleasingly practical:

1. Run the frontier model in production and log inputs and outputs for a target task.
2. Curate a dataset from those logs — a few thousand good examples is often enough for a narrow task.
3. Fine-tune a small open-weights model on it (LoRA or QLoRA, so it's affordable).
4. Evaluate the fine-tuned model against the frontier model on held-out real examples.
5. Route that task class to it, keeping escalation to the frontier model for low-confidence cases.

For narrow, well-defined tasks, a fine-tuned small model regularly matches or beats a general frontier model — it has seen thousands of examples of exactly your task and your label taxonomy, which no amount of prompting conveys. The catch is that it's now your model: you own its evaluation, its drift, its retraining, and its serving. That's a real ongoing cost, justified by volume and not by elegance.

The Non-Cost Reasons

Cost dominates the conversation, but two other drivers are often the actual decision:

Privacy and data residency. Healthcare records, legal documents, source code under NDA, EU data-residency obligations — for some workloads, no API price makes sending the data acceptable. Local inference is the only architecture that satisfies the requirement, and the cost comparison never happens.

Latency floor. A network round trip to a hosted API has an irreducible floor of tens to hundreds of milliseconds before the model does anything. A quantized small model on local hardware can beat that meaningfully. For interactive experiences — IDE completions, live transcription, on-device assistants — that difference is the product.

A Decision Path That Works

The sequence I'd actually follow, in order, without skipping steps:

1. Ship on a frontier API. Prove the feature is worth optimizing before optimizing it. Most features die before this matters.
2. Instrument cost and latency per request class. You cannot optimize what you haven't measured, and the distribution is always more skewed than people guess.
3. Build an eval set from real traffic. Every subsequent decision depends on it.
4. Route the easy classes to a cheap hosted model. Biggest win, smallest effort, no infrastructure. Many teams should stop here.
5. Fine-tune a small model for your highest-volume narrow task, if volume justifies owning a model.
6. Self-host only when volume is high and steady enough to keep GPUs busy, or when data residency forces it.

Most teams can stop at step 4 and capture the majority of the available savings. Teams that jump straight to step 6 because self-hosting sounds cheaper usually discover that idle GPUs and the engineering time to keep a serving stack healthy cost more than the API bill they were avoiding. The cheapest inference is still the call you didn't have to make — which is why semantic caching and a hard look at whether a step needs a model at all belong above all of this on the list.

---

Go deeper: the Gen-AI roadmap covers the surrounding material, including cost optimization and model routing, evaluation frameworks, and tokens and context windows.