Pre-training, Post-training & Inference
The three-stage model lifecycle — and why nothing in it directly rewards being correct, which is the actual, mechanical reason LLMs hallucinate.
Back to the PM's Question
Two guides ago, a support bot confidently invented a "special 2024 refund policy" that never existed, and a PM asked the obvious question: "Doesn't it just know the answer? Why would it make something up?" The previous two guides covered how a model represents and relates text — attention, tokens, embeddings, context windows. None of that explains hallucination on its own. This guide does: a model isn't built in one step, it goes through three distinct stages, and the fabricated policy is a direct, predictable consequence of how the first two of them work.
Stage 1 — Pre-training: Learning to Predict, Not to Be Correct
Pre-training is where the vast majority of compute goes: the base model reads an enormous corpus — trillions of tokens of web text, books, code, and more — and is trained on nothing but "predict the next token." Critically, nothing in this objective rewards factual accuracy directly. The model is rewarded for producing plausible continuations of text, and most of the time, plausible and factually correct are the same thing, because most training text is actually true. But "plausible" and "true" are not the same property, and when they diverge — an obscure fact, a specific policy number, a citation — a model trained purely to predict likely-sounding text will produce likely-sounding text regardless of whether it's grounded in anything real. That's not a bug introduced later in the pipeline. It's the base objective working exactly as designed, applied to a question it doesn't actually have a verified answer for.
A model straight out of pre-training — a base model — isn't even conversational by default. Ask it a question and it's just as likely to continue the pattern of a question (writing three more related questions, the way a quiz document would) as it is to answer it. Getting from that raw completion engine to something that reliably acts like a helpful assistant is the job of the next stage.
For a sense of just how far "predict the next token, at scale" goes before any post-training is applied, Language Models are Few-Shot Learners (the GPT-3 paper) is the one that convinced most of the field that scale alone produces qualitatively new capabilities, not just quantitatively better ones.
Stage 2 — Post-training: Teaching the Model to Be Useful, Not Just Fluent
Post-training reshapes the base model's behavior without fundamentally changing what it learned about language during pre-training. It typically happens in a couple of steps:
- Supervised Fine-Tuning (SFT): the model is trained on a curated set of high-quality example conversations — instruction in, ideal response out — which teaches it the shape of being an assistant: answering directly, following formatting instructions, refusing certain requests.
- RLHF (Reinforcement Learning from Human Feedback): human raters (or a trained reward model standing in for them) compare pairs of candidate responses and indicate which one is better; the model is then optimized against that reward signal. OpenAI's Training Language Models to Follow Instructions with Human Feedback (the InstructGPT paper) is the one that put this pipeline on the map.
- DPO (Direct Preference Optimization): a newer, simpler alternative to RLHF that skips training a separate reward model entirely, optimizing directly on the preference pairs instead — introduced in Direct Preference Optimization (2023) and now widely used because it's cheaper and more stable to train than full RLHF.
Either way, this is where a lot of what people call "alignment" comes from — helpfulness, refusing harmful requests, matching a particular tone.
Hugging Face's Illustrating Reinforcement Learning from Human Feedback is the clearest walkthrough of the reward-model-plus-RL pipeline if you want the mechanics without reading the full papers first.
Here's the part that connects directly back to the hallucinated refund policy: RLHF and DPO both optimize for responses that human raters rated as good — typically meaning confident, well-structured, plausible-sounding, and unhelpfully-refusal-free. None of those qualities require the underlying claim to be true, and a rating process built on "does this response look helpful" will happily reward a confidently wrong answer over a hedgy correct one, unless raters are specifically trained and instructed to penalize unsupported claims. Post-training makes a model sound authoritative. It does not, by itself, make the model check its work. That gap between "sounds right" and "is right" is exactly what RAG, grounding, and citation-verification (Phase 4 onward) exist to close — post-training alone was never going to solve it.
A subtlety worth internalizing early: RLHF can actively make hallucination harder to spot, not just fail to prevent it. A model punished for hedging and rewarded for confident, fluent answers learns to sound equally confident whether it knows the answer or is fabricating one. Fluency is not a truth signal.
Stage 3 — Inference: Where Temperature and Sampling Live
Everything above happens once, during training. Inference is what happens every single time you send a prompt: the trained model computes a probability distribution over every possible next token, and something has to decide which token actually gets picked. That "something" is controlled by sampling parameters you set on every API call:
| Parameter | What it controls | Low value | High value |
|---|---|---|---|
| Temperature | How much the model deviates from picking the single most-likely next token | Deterministic, focused, "boring" — good for classification, extraction, code | Varied, exploratory — good for brainstorming, creative writing |
| top_p (nucleus sampling) | Restricts sampling to the smallest set of tokens whose combined probability crosses a threshold | Narrow, safe candidate pool | Wider candidate pool, more variety |
| top_k | Caps the candidate pool to a fixed number of the k most-likely tokens | Very restricted (e.g. top 10) | Looser restriction (e.g. top 100) |
Most teams tune one of these — usually temperature — and leave the rest at their defaults. Turning all three at once mostly just makes behavior harder to reason about without a proportional benefit.
Temperature 0 does not guarantee byte-for-byte identical output across calls, even for the exact same prompt. Floating-point non-determinism in how providers batch and execute requests means "deterministic" is closer to "very consistent" in practice. Don't build logic — cache keys, dedup checks, anything — that assumes perfectly reproducible output, even at temperature 0.
There's a fourth, newer wrinkle worth knowing about: reasoning ("thinking") models, which spend additional inference-time tokens working through a problem step by step in a hidden or semi-visible scratch space before committing to a final answer. The underlying idea traces back to Chain-of-Thought Prompting Elicits Reasoning in Large Language Models (2022), which first showed that getting a model to write out intermediate steps — even just via prompting, before any special training existed for it — measurably improved multi-step problem solving. Today's reasoning models bake that behavior in through post-training rather than relying on the user to prompt for it. Either way, it measurably improves performance on multi-step problems — math, complex debugging, multi-hop questions — at the direct cost of latency and token spend, which is exactly the kind of tradeoff you'll be making deliberately once you get to model routing and cost optimization in Phase 9.
Check yourself
A model confidently states a specific policy that turns out not to exist. Based on the three-stage lifecycle, what's the most accurate explanation?
Back to the Support Bot
That fabricated "2024 refund policy" now has a fully mechanical explanation, not a mysterious one: a decoder-only Transformer, pre-trained to predict plausible next tokens over a huge corpus that never included this company's actual policy, post-trained via RLHF to sound confident and helpful, sampling at inference time from a probability distribution with no built-in fact-checking step. Every stage did exactly what it was trained to do. The gap was never a "bug" in any single component — it was a missing layer that none of these three stages was ever designed to provide: verification against ground truth.
That's not a dead end — it's the on-ramp to the rest of this roadmap. Phase 2 covers how to constrain and structure what a model produces through prompting. Phase 3 covers managing what actually goes into the context window across a long session. And Phase 4 introduces RAG — the technique that was, not coincidentally, the actual fix the team in this story reached for: forcing the model to ground its claims in retrieved, verifiable documents instead of whatever pattern-matched plausibly from its training data.
If you only take one thing from this phase into the rest of the roadmap, make it this: every technique from here forward — prompting, RAG, agents, guardrails — is fundamentally about compensating for the fact that a next-token predictor has no innate concept of truth. Once that clicks, the reason each technique exists stops feeling arbitrary.
Frequently asked questions
Does RLHF mean the model has been explicitly taught which facts are true?
No — RLHF shapes response style and behavior (helpfulness, tone, refusals) based on human preference comparisons, not a fact-verification process. A model can be extremely well-aligned by RLHF standards — polite, well-structured, appropriately cautious about clearly dangerous requests — and still confidently state something false, because truthfulness was never the thing being directly optimized.
Should I always use temperature 0 to avoid hallucination?
Temperature 0 makes output more consistent, not more truthful — it just makes the model pick its single most-likely next token instead of sampling. If the most-likely continuation is a plausible-sounding fabrication, temperature 0 will produce that fabrication reliably instead of only sometimes. Reducing hallucination requires grounding (Phase 4), not just lowering temperature.
Is DPO strictly better than RLHF?
Not universally — DPO is simpler and cheaper to train because it skips the separate reward-model step, which is why it's become popular. RLHF's explicit reward model can, in some setups, be reused or inspected in ways a DPO-trained model can't. Most applied AI engineers won't be training either directly, but it's worth knowing both exist and roughly why a provider might pick one.