Tokens, Context Windows & Latent Space
What a token actually is, why context windows are a hard ceiling rather than a soft suggestion, and how embeddings turn meaning into geometry.
The Unit Everything Is Measured In
The last guide covered how a model relates tokens to each other through attention. This one covers what a token actually is, what happens before attention ever runs, and the hard limit that governs how much of a conversation a model can actually see — the mechanic behind a very common production complaint: "the bot forgot something the user told it earlier."
A token is the model's actual unit of work — not quite a word, not quite a character. "unbelievable" typically splits into something like un, believ, able. Common short words often stay whole; rare words, typos, and non-English text fragment into more pieces. Most modern tokenizers do this using Byte-Pair Encoding (BPE), a compression-derived algorithm adapted for language models in Neural Machine Translation of Rare Words with Subword Units — worth a skim if you've ever wondered why tokenization looks the way it does instead of just splitting on whitespace.
This isn't trivia — it has three direct consequences you'll run into constantly:
- You pay per token, not per word or per character. Every hosted provider bills separately for input tokens (what you send) and output tokens (what comes back), and they're usually priced differently.
- The model has a hard limit on how much text it can see at once — the context window, measured in tokens. Go over it and the oldest content simply isn't visible to the model anymore.
- Generation speed scales with output tokens. A 2,000-token answer takes meaningfully longer to produce than a 50-token one, because the model generates output one token at a time, each one depending on everything generated before it.
Rough rule of thumb for English text: 1 token ≈ 4 characters ≈ ¾ of a word. A 1,000-word blog post lands around 1,300–1,400 tokens. You'll do this mental math constantly once you start estimating cost and context usage, so it's worth internalizing now rather than looking it up every time.
Latent Space: Where "Meaning" Actually Lives
Attention relates tokens to each other, but before any of that happens, every token first gets converted into a vector of numbers — an embedding — typically somewhere between a few hundred and a few thousand dimensions. This numeric representation, and the high-dimensional space it lives in, is what people mean by latent space.
The property that makes this useful: proximity in that space corresponds to semantic similarity. The embedding for "puppy" sits close to the embedding for "dog," which sits much farther from the embedding for "quarterly earnings report." No one programmed that geometry by hand — it fell out of training the model to predict text well, because words that mean similar things tend to appear in similar surrounding contexts, and the model's internal representations gradually organized themselves to reflect that. This idea — that meaning can be represented as position in a vector space, learned purely from how words co-occur — goes back to Efficient Estimation of Word Representations in Vector Space (Mikolov et al., 2013, the word2vec paper), and every embedding model in production today is a direct descendant of that observation.
This is not a side effect you only care about in theory. It's the entire mechanism behind semantic search and RAG (retrieval-augmented generation), which is the whole subject of Phase 4 of this roadmap: if you embed a user's question and embed a million documents into that same latent space, "nearby in the vector space" becomes a genuinely useful proxy for "relevant to the question" — even when the question and the document don't share a single word in common.
Two things worth trying yourself before moving on: the OpenAI Tokenizer tool lets you paste any text and watch it split into tokens in real time — the fastest way to build intuition for the "1 token ≈ ¾ word" rule of thumb. And the TensorFlow Embedding Projector lets you rotate a real, high-dimensional embedding space in 3D and watch semantically related words cluster together — "latent space" stops being an abstract phrase the moment you can spin it around with a mouse.
The Context Window Is a Hard Ceiling, Not a Soft Suggestion
The context window is the total number of tokens — input plus output combined — a model can process in a single call. Context windows have grown dramatically over the life of this field, from a few thousand tokens in early chat models to hundreds of thousands of tokens (and beyond) in current frontier models. But "large" is not "infinite," and the failure mode when you exceed it is easy to misdiagnose the first time you hit it: the oldest messages in a long conversation don't get "compressed" or "summarized" automatically — they simply fall outside the window and the model never sees them again. If a user references something they told the bot forty messages ago and that content happened to fall outside the window, the model isn't being forgetful in any humanlike sense; it's mechanically missing data.
There's a second, subtler failure mode worth knowing about even this early: a bigger window doesn't mean a model uses everything inside it equally well. Lost in the Middle: How Language Models Use Long Contexts found that models are noticeably better at recalling information from the start and end of a long context than from the middle — a real, measured effect, not a hypothetical one. This is exactly the problem Phase 3 of this roadmap — context engineering — exists to manage deliberately, instead of hoping it never comes up.
Check yourself
A support bot suddenly 'forgets' something a user mentioned 45 messages ago in a long conversation. What's the most likely mechanical cause?
What's Next
Tokens, embeddings, and context windows explain how text gets represented and how much of it a model can hold onto — but not yet why a model sometimes states things confidently that simply aren't true. That's the last piece: what actually happens during training, and why nothing in that process directly rewards being correct.
Frequently asked questions
Why do different providers count tokens differently for the same text?
Each model family uses its own tokenizer, trained on its own vocabulary. GPT, Claude, and Gemini models all split the same sentence into slightly different tokens, which is why the same prompt can cost a different number of tokens depending on which provider you send it to. Always check token counts using that specific provider's tokenizer, not a generic estimate, when cost precision matters.
Is a bigger context window always better?
Not automatically. A larger window means a model can technically see more, but retrieval quality from the middle of a very long context is empirically worse than from the beginning or end — a phenomenon often called 'context rot,' which Phase 3 covers in depth. A bigger window raises the ceiling; it doesn't guarantee the model uses everything under that ceiling equally well.
Are embeddings the same for every provider and model?
No. Each embedding model defines its own vector space, trained on its own data — an embedding from one model isn't directly comparable to an embedding from another. If you switch embedding models in a RAG pipeline, you generally need to re-embed your entire document set, not just new documents going forward.