The model never sees your words.
It sees the numbers your words were translated into.
Tokenization is that translation.
And like any translation,
something is always lost — or gained — in the process.
01 — Start With an Analogy
Imagine you want to teach a calculator to understand a recipe. The problem: calculators don't understand "a pinch of salt" or "two cups of flour." They only understand numbers. So before the calculator can do anything useful, someone has to translate every ingredient into a code from a fixed list — flour = 12, salt = 47, eggs = 8.
That fixed list is the vocabulary. The translation process is tokenization. The numbered codes are tokens.
Language models work exactly the same way. They are, at their core, mathematical functions — matrix multiplications, additions, dot products. None of those operations work on letters or words. So before your message ever reaches the model, a tokenizer converts it into a sequence of numbers. Only then does the model start "thinking."
The model never sees the word "cat." It sees the number 9246. It never sees "running" — it sees 8996. Every word, every punctuation mark, every space has been replaced by a number from a predefined list before the model processes anything. Tokenization is the bridge between human language and machine math.
02 — The Vocabulary, the Token, and the Pipeline
Let's define the core concepts precisely — in plain English first, then with the technical layer underneath for those who want it.
Here is the full journey your text takes before the model sees it:
Step 1: "I love AI" → split into tokens → ["I", " love", " AI"]
Step 2: Each token gets its ID → [40, 1842, 9552]
Step 3: Each ID is looked up in a table → a vector of ~1,000 numbers
Step 4: Those vectors enter the model — now the math begins
The model never touches raw text. It only ever sees vectors of numbers.
Each token ID is an integer index into an embedding matrix of shape |V| × d_model. For GPT-2: 50,257 × 768 = ~38.6M parameters just in the embedding layer. The embedding lookup is a simple table read — E[token_id] — returning a dense vector in ℝd. This vector is what actually flows through the transformer's attention and feed-forward layers. The vocabulary size |V| directly determines this matrix's size and is one of the most consequential hyperparameters in model design.
03 — Three Ways to Slice Text
Once we agree that text must become a sequence of numbered chunks, the next question is: how big should each chunk be? Should each letter be its own chunk? Each word? Something in between? There are three approaches, each with its own tradeoffs.
Strategy 1 — One Letter at a Time
The simplest idea: every single letter (character) is its own token. The word "cat" becomes three tokens: c, a, t.
c+a+t = a furry animal, with no direct signal that these three letters belong together as a concept.Strategy 2 — One Word at a Time
The intuitive approach: split on spaces, and each word is its own token. "I love AI" becomes three tokens: I, love, AI.
[UNK] token. The model is completely blind to it. Also, related words like "run", "running", and "ran" are treated as three completely unrelated entries, with nothing connecting them.Strategy 3 — Something In Between (Subword)
The modern solution, used by every major AI model today. The idea: common words get their own token, but rare or long words are split into smaller meaningful pieces. It's the best of both worlds.
Think of it like a language with prefixes and suffixes. The model knows "token" as a concept and "ization" as a suffix meaning "the process of." So "tokenization" is immediately understood as "the process of making tokens" — without ever having a specific entry for it. New words are no longer a problem: they're just combinations of known building blocks. And the vocabulary stays a manageable size — typically 32,000 to 128,000 entries — instead of needing millions of word entries.
The Tradeoff at a Glance
| Strategy | Chunk Size | Vocab Size | Sequence Length | Unknown Words? |
|---|---|---|---|---|
| Character | One letter | Tiny (~256) | Very long | Never |
| Word | One word | Huge (100K+) | Short | Constantly |
| Subword | Word pieces | Balanced (32K–128K) | Moderate | Never |
This is why every modern language model — GPT, BERT, LLaMA, Gemma — uses subword tokenization. The exact algorithm used to decide which word pieces to include in the vocabulary differs between models, and that's the topic of Day 24: Tokenization Deep Dive.
04 — Special Tokens: The Punctuation of AI Thought
Beyond the regular text tokens, every model reserves a small set of special tokens — entries in the vocabulary that don't represent any real word, but instead carry a structural instruction. Think of them as punctuation marks for the model's internal language.
Here is what each one does, in plain terms:
[CLS] — "I am the summary"[SEP] — "New section starts here"[PAD] — "Ignore this"[UNK] — "I don't know this"<|endoftext|> — "I'm done"<s> / </s> — "Start / End"<s> is placed at the beginning of every input — it's the model's "ready" signal. </s> is placed at the end — it means "stop here." They serve the same purpose as [CLS]/[SEP] and <|endoftext|> in other architectures.Special tokens are tied to specific model architectures. BERT's [CLS] is token ID 101. LLaMA's token ID 101 is something completely different. If you accidentally use BERT's tokenizer with LLaMA's weights, every single token ID maps to the wrong entry in the model's memory table — the model receives total gibberish and outputs total gibberish. The tokenizer and the model are inseparable. Always load them together from the same source.
05 — Why Tokenization Matters in the Real World
Tokenization isn't just a technical detail that happens behind the scenes — it has direct, visible consequences that affect every person who uses an AI model.
Token Limits Are Not Word Limits
When an AI service says "this model supports 128,000 tokens," many people assume that means 128,000 words. It doesn't. In English, one word is roughly 1.3 tokens on average — so 128,000 tokens ≈ 98,000 words. But more importantly, this ratio changes completely depending on the language you're writing in.
This happens because the vocabulary of most early LLMs was built predominantly from English text. English words and word-pieces are well-represented; Arabic morphemes and Chinese characters are not. The model has to use many more small pieces to express the same idea in Arabic.
This isn't just a cost issue — it affects quality too. When Arabic text is represented as 4× more tokens than equivalent English text, the model's "attention" is spread thinner. Relationships between words that are conceptually close may end up far apart in the token sequence. This is one of the real, measurable reasons why AI models tend to perform better on English than on other languages — and it starts here, at tokenization. Day 24 covers exactly how modern models like LLaMA 3 have addressed this with larger, more multilingual vocabularies.
Why "ChatGPT Can't Count Letters"
You may have seen the famous example: ask ChatGPT "how many R's are in the word STRAWBERRY?" and it sometimes says 2. There are 3. This isn't the model being stupid — it's a tokenization consequence.
This also explains why LLMs struggle with certain spelling tasks, rhyming, and letter-by-letter manipulation — the token is the atom of the model's perception, and what's inside an atom is invisible to it.
Counting Tokens Before You Pay
Most AI APIs charge per token. Knowing this, a few practical habits save real money:
🔗 Tiktokenizer — Interactive tokenizer visualizer
Type any text and watch in real time how it gets split into tokens under GPT-4's tokenizer. Each token is colour-coded. The fastest way to build genuine intuition for what tokenization actually does to your words. Try "strawberry", then try the same word in Arabic.
🔗 OpenAI Tokenizer — Official playground
OpenAI's own tool for visualising tokenization and counting tokens for GPT-3.5 and GPT-4. Useful for estimating API costs before building a production system.
📄 Neural Machine Translation of Rare Words with Subword Units — Sennrich et al. (2016)
The paper that brought subword tokenization into mainstream NLP. A short, readable read that clearly shows why breaking words into pieces beats treating each word as atomic — especially for rare and unseen words.
- Text must become numbers. Language models are mathematical functions — they can't process strings. Tokenization converts every word, punctuation mark, and space into a number (token ID) before the model ever sees it.
- The vocabulary is the model's dictionary. A fixed list of 32,000–128,000 entries, each with a unique ID. If something isn't in the vocabulary, it gets broken into smaller pieces that are — no word is ever truly "unknown" in a modern model.
- Three strategies: letter, word, or piece. Character-level (tiny vocab, very long sequences), word-level (short sequences, but fails on unknown words), and subword (the modern winner — compact vocabulary, no unknown words, handles any language).
- Special tokens are structural signals. [CLS] means "summarise this whole sequence," [PAD] means "ignore this slot," <|endoftext|> means "I'm done generating." They are architecture-specific — mixing a tokenizer from one model with another model's weights produces gibberish.
- Token counts ≠ word counts. English averages ~1.3 tokens per word, but Arabic can reach 4–6. The same context window holds far less non-English content — this has real cost and quality implications for multilingual applications.
- Tokens are atomic. The model cannot see inside a token. This is why LLMs sometimes fail at letter-counting, spelling, and other tasks that require character-level awareness — the token is the smallest unit the model perceives.
- What is tokenization and why is it necessary?
Tokenization is the process of converting raw text into a sequence of integer token IDs before it enters a language model. It is necessary because neural networks are mathematical functions — they compute dot products, matrix multiplications, and additions, none of which are defined on strings. Every character, word, and punctuation mark must be mapped to a number from a fixed vocabulary before the model can process it. The tokenizer bridges the gap between human language and machine math, and its design has direct downstream effects on sequence length, vocabulary coverage, multilingual capability, and even the types of tasks the model performs well or poorly on. - Why is subword tokenization the dominant approach in modern LLMs?
Subword tokenization solves the two main failures of the simpler strategies. Character-level tokenization requires no vocabulary but produces extremely long sequences — a short paragraph can become thousands of tokens, making processing slow and expensive. Word-level tokenization keeps sequences short but requires a massive vocabulary and fails completely on any unseen word, mapping it to a meaningless [UNK] token. Subword sits between both extremes: a vocabulary of 32K–128K word pieces can represent any string without unknown tokens (by breaking new words into known fragments), while keeping sequence lengths manageable. It also gives the model a natural sensitivity to morphology — "run", "running", and "runner" share the "run" token, providing a built-in signal that these words are related. - What are special tokens, and what happens if you use the wrong tokenizer with a model?
Special tokens are reserved vocabulary entries with structural meaning that are never generated by the tokenization algorithm itself — they are added manually. Examples include [CLS] (signals the model to produce a sequence-level summary for classification), [PAD] (fills shorter sequences in a batch to equal length), [SEP] (marks segment boundaries), and <|endoftext|> (GPT's stop signal). They are completely architecture-specific — BERT, GPT, and LLaMA each define different special tokens at different token IDs. If you use BERT's tokenizer with LLaMA's model weights, token ID 101 (which BERT uses for [CLS]) will be looked up in LLaMA's embedding table, where it points to a completely different, unrelated token. Every single token in the input is wrong from the model's perspective. The output is garbage. The tokenizer and model checkpoint are inseparable components of the same system and must always be loaded together.