Tokenization Deep Dive:
BPE, WordPiece & SentencePiece

A language model never reads a word.
It reads numbers — integers pointing to entries in a lookup table.
The algorithm that converts your sentence into those numbers
is the tokenizer.

Change the tokenizer and you change everything:
what the model can represent, how much context it can hold,
how fairly it treats different languages.

Tokenization is where language meets math.
It deserves more attention than it gets.

01 — Why Not Just Split on Spaces?

The most obvious approach to tokenization is word-level splitting: one token per word. "The cat sat" → ["The", "cat", "sat"]. Simple, interpretable — and broken in four specific ways that matter enormously for large-scale language models.

The four failure modes of word-level tokenization:

The solution, developed independently by several teams between 2015 and 2018, is subword tokenization: split text at the level of meaningful character sequences, not words. Keep common words whole. Break rare words into smaller reusable pieces. Never produce an unknown token, because at the absolute minimum you can fall back to individual characters or bytes. Three algorithms dominate: BPE, WordPiece, and SentencePiece.

02 — Byte-Pair Encoding (BPE): How GPT Tokenizes

BPE (Sennrich et al., 2016) was originally a data compression algorithm from 1994, repurposed for NLP. The key idea: start with individual characters, then iteratively merge the most frequent adjacent pair until you reach your target vocabulary size.

BPE Algorithm: Step by Step

Let's trace BPE on a tiny corpus: "low low low lowest newer newer" (word frequencies matter, not just words).

Initial state: split every word into characters + end-of-word marker </w>. Count all adjacent pairs.

This continues until the vocabulary reaches the target size. The algorithm produces a merge table — an ordered list of merge rules. At inference time, you apply the same merges in the same order to tokenize new text. The result is deterministic: the same string always produces the same token sequence.

Byte-Level BPE: How GPT-2 and GPT-4 Do It

GPT-2 introduced a crucial improvement: byte-level BPE. Instead of starting with Unicode characters, it starts with the 256 possible bytes. This guarantees the tokenizer can handle any text in any language, any encoding, any emoji — because every string is ultimately a sequence of bytes. The vocabulary never needs an [UNK] token. "مرحبا" (Arabic for "hello"), 你好 (Chinese), and 🌙 are all representable without special handling.

GPT-2/3/4 and LLaMA-2 all use byte-level BPE. The vocabulary sizes:

Worked Example: Tokenizing "unhappiness"

With a GPT-style BPE tokenizer trained on English text:

3 tokens. The splits follow frequency — "happ" and "iness" are common English substrings that appeared often enough to get merged into single tokens during BPE training.

Compare the same word in different tokenizers:

This directly affects cost: GPT-4 charges per token. A document tokenized into 1,000 tokens by cl100k might tokenize into 1,400 tokens with LLaMA-2's smaller vocabulary — meaning the same text consumes 40% more context window and costs 40% more to process.

03 — WordPiece: How BERT Tokenizes

WordPiece (Schuster & Nakamura, 2012; used in BERT since 2018) is superficially similar to BPE but makes a fundamentally different choice: instead of merging the most frequent pair, it merges the pair that maximises the language model likelihood of the training data.

The Core Difference from BPE

Why does this matter? BPE will eagerly merge very common characters like "e" and "s" because they appear together often — even if they always appear as parts of different words and the merge isn't linguistically meaningful. WordPiece is more conservative: it only merges pairs that are more likely together than apart, which tends to produce more linguistically coherent subwords.

The ## Continuation Marker

WordPiece marks subword pieces that continue a word with a ## prefix. This is a crucial design choice: it preserves the information about whether a piece starts a new word or continues one.

The ## prefix means "I am attached to the previous token — I don't start a new word." This lets the model's attention mechanism distinguish "play" at the start of a word from "##play" in the middle, even though they map to different token IDs. Without this marker, the model might confuse "play" in "playing" with "play" as a standalone word.

WordPiece Vocabulary Construction

WordPiece starts with all individual Unicode characters in the training data (or a fixed set of bytes), then greedily merges pairs using the PMI criterion. BERT-base uses a vocabulary of 30,522 tokens — smaller than GPT-2's 50,257 because BERT was designed for shorter inputs and understanding tasks rather than open-domain generation. The vocabulary includes:

• 100 unused slots (reserved for domain-specific fine-tuning)
• Special tokens: [PAD], [UNK], [CLS], [SEP], [MASK]
• All printable ASCII characters individually
• ~30,000 subword pieces learned from the training corpus

04 — SentencePiece: How LLaMA & T5 Tokenize

SentencePiece (Kudo & Richardson, 2018) is not a tokenization algorithm in itself — it's a tokenization framework that can run either BPE or Unigram as the underlying algorithm, while solving two problems neither BPE nor WordPiece address well: language independence and whitespace handling.

The Two Key Innovations

Unigram Language Model — SentencePiece's Alternative Algorithm

SentencePiece can use either BPE or a second algorithm called Unigram. Instead of starting with characters and merging up, Unigram starts with a large vocabulary and prunes it down.

The key advantage of Unigram over BPE: it produces probabilistic tokenization. A given text can be tokenized in multiple valid ways, each with a probability. During training, you can sample from this distribution — exposing the model to multiple segmentations of the same text — which improves robustness and generalisation. BPE is deterministic; Unigram is not.

Who Uses SentencePiece

05 — Worked Comparison: The Same Sentence Across Four Tokenizers

Let's tokenize one sentence through four real tokenizers and see exactly what happens:

"The LLM tokenized the word 'تعلّم' into 5 tokens."

GPT-4 (cl100k BPE, 100k vocab):

17 tokens total — Arabic characters split to individual bytes/chars since Arabic text was rare in GPT-4's training data relative to vocabulary budget

BERT (WordPiece, 30k vocab):

14 tokens — Arabic word collapsed to [UNK] (BERT's base vocabulary is English-focused); "LLM" split as "LL" + "##M"

LLaMA-2 (BPE via SentencePiece, 32k vocab):

18 tokens — Arabic is byte-level but handled without [UNK]; space encoded as ▁ prefix in each token

mT5 (Unigram via SentencePiece, 250k vocab — multilingual):

12 tokens — 250k multilingual vocab keeps Arabic word whole; "tokenized" is one token; most efficient of the four

06 — Tokenization and Fairness: The Multilingual Gap

One of the most important — and often overlooked — consequences of tokenizer design is linguistic fairness. A tokenizer trained predominantly on English text will produce very different token counts for equivalent texts in other languages.

Translating "I love machine learning" into several languages and counting tokens with GPT-3.5's tokenizer (cl100k):

An Arabic user expressing the same thought pays 3.25× more tokens — meaning 3.25× less context window and 3.25× higher API cost for the same amount of meaning. This is not a trivial inconvenience — it makes LLMs practically unusable for many languages at the same cost point as English. Multilingual models like mT5 and LLaMA-3 (with its 128k vocabulary) specifically address this by allocating more vocabulary budget to non-English scripts.

07 — Special Tokens: The Control Layer

Every tokenizer reserves a set of special tokens that carry structural meaning for the model. These are not learned from text — they are inserted by the tokenizer to mark boundaries, roles, and instructions.

Special tokens are critical for chat models. The ChatML format used by GPT-4 structures every conversation as:

The model learned during fine-tuning to produce a response after seeing <|im_start|>assistant. The special tokens are literally the format the model was trained to follow.

08 — Head-to-Head: BPE vs WordPiece vs SentencePiece

09 — Code: BPE From Scratch + HuggingFace Tokenizers

A minimal BPE implementation in pure Python — the same algorithm that GPT-2 uses under the hood — followed by working HuggingFace tokenizer examples.

from collections import defaultdict, Counter
import re

# ─────────────────────────────────────────────────────────────────────────
# Minimal BPE implementation from scratch
# ─────────────────────────────────────────────────────────────────────────

def get_vocab(corpus: list[str]) -> dict:
    """Convert corpus to character-level vocab with word frequencies."""
    vocab = defaultdict(int)
    for word in corpus:
        # Represent each word as tuple of chars + end-of-word marker
        vocab[' '.join(list(word)) + ' </w>'] += 1
    return dict(vocab)


def get_pairs(vocab: dict) -> Counter:
    """Count all adjacent symbol pairs across the vocabulary."""
    pairs = Counter()
    for word, freq in vocab.items():
        symbols = word.split()
        for i in range(len(symbols) - 1):
            pairs[(symbols[i], symbols[i+1])] += freq
    return pairs


def merge_pair(pair: tuple, vocab: dict) -> dict:
    """Merge all occurrences of a pair in the vocabulary."""
    new_vocab = {}
    bigram = ' '.join(pair)
    replacement = ''.join(pair)
    for word in vocab:
        new_word = word.replace(bigram, replacement)
        new_vocab[new_word] = vocab[word]
    return new_vocab


def train_bpe(corpus: list[str], num_merges: int) -> tuple[dict, list]:
    """
    Train BPE on a corpus.
    Returns final vocab and the ordered list of merge rules.
    """
    vocab = get_vocab(corpus)
    merges = []

    print(f"Initial vocab size: {len(set(' '.join(vocab.keys()).split()))}")

    for i in range(num_merges):
        pairs = get_pairs(vocab)
        if not pairs:
            break
        best = pairs.most_common(1)[0][0]   # most frequent pair
        vocab = merge_pair(best, vocab)
        merges.append(best)
        if i < 5:   # show first 5 merges
            print(f"  Merge {i+1}: {best[0]!r} + {best[1]!r} → {best[0]+best[1]!r}  (freq={pairs[best]})")

    return vocab, merges


def tokenize_bpe(word: str, merges: list) -> list[str]:
    """Apply learned BPE merge rules to tokenize a new word."""
    tokens = list(word) + ['</w>']
    for merge_pair in merges:
        i = 0
        while i < len(tokens) - 1:
            if (tokens[i], tokens[i+1]) == merge_pair:
                tokens = tokens[:i] + [merge_pair[0] + merge_pair[1]] + tokens[i+2:]
            else:
                i += 1
    return tokens


# Train on a small corpus
corpus = (["low"] * 5 + ["lowest"] * 2 + ["newer"] * 6 +
          ["wider"] * 3 + ["new"] * 8 + ["play"] * 4 +
          ["playing"] * 5 + ["played"] * 3 + ["player"] * 2)

vocab, merges = train_bpe(corpus, num_merges=15)

print(f"\nFinal vocab (top entries):")
for k, v in sorted(vocab.items(), key=lambda x: -x[1])[:5]:
    print(f"  {k!r}: {v}")

print(f"\nTokenize new words using learned merges:")
for word in ["newer", "playing", "lowest", "newness"]:
    toks = tokenize_bpe(word, merges)
    print(f"  {word:12s} → {toks}")

# ─────────────────────────────────────────────────────────────────────────
# HuggingFace tokenizers — comparing BPE, WordPiece, SentencePiece
# pip install transformers
# ─────────────────────────────────────────────────────────────────────────
from transformers import AutoTokenizer

test_sentences = [
    "unhappiness is a temporary state",
    "def tokenize(text: str) -> list[int]:",
    "مرحباً بكم في التعلم الآلي",   # Arabic: "Welcome to machine learning"
]

tokenizers = {
    "GPT-2 (BPE 50k)":         "gpt2",
    "BERT-base (WordPiece 30k)": "bert-base-uncased",
    "LLaMA-2 (SentPiece 32k)":  "meta-llama/Llama-2-7b-hf",
}

for name, model_id in tokenizers.items():
    tok = AutoTokenizer.from_pretrained(model_id)
    print(f"\n{'─'*60}")
    print(f"{name}")
    for sent in test_sentences:
        tokens = tok.tokenize(sent)
        ids    = tok.encode(sent)
        print(f"  [{len(tokens):2d} tok] {tokens}")

10 — Why This Matters in Practice

Token count = cost + context. Every LLM API charges per token. A sentence in Arabic costs 3–5× more tokens than the same sentence in English on most tokenizers. Code is also expensive: symbols like (, ), ->, [ often each get their own token, so a function signature like def f(x: int) -> str: might use 14 tokens even though it's only 22 characters — about 1.6 characters per token instead of the typical 4.

Tokenizer mismatch causes subtle bugs. If you fine-tune a model with one tokenizer and run inference with another, every token ID maps to a different piece of text. The model produces garbage. This happens more often than you'd expect when swapping base models during development. Always verify tokenizer.vocab_size and a few sample encodings before fine-tuning.

Some tasks need careful token counting. When building RAG pipelines (Day 29), you chunk documents to fit within a context window. If your chunk splitter uses character counts but your model uses token counts, you'll miscalculate and either truncate important context or waste budget. Always measure chunk size in tokens, not characters, and use the model's actual tokenizer for the measurement.

Tokenization reveals model boundaries. GPT-4 tokenizes SolidGoldMagikarp as a single token — a Reddit username that appeared frequently enough in training data to get its own vocabulary entry. There are entire vocabulary slots reserved for tokens that look like random strings but are actually subreddit names, usernames, and domain-specific jargon from the training corpus. Studying your tokenizer's vocabulary is one way to understand what's in your training data.

11 — A Thought to Sit With

12 — Coming Up Next

Tomorrow — Day 28: Fine-Tuning LLMs: Full, LoRA & QLoRA. We've now built the complete transformer, added positional encodings, understood BERT vs GPT, and seen how text becomes tokens. Day 28 answers the practical question every ML engineer faces: you have a pre-trained LLM and a specific task — how do you adapt it? Full fine-tuning, LoRA's low-rank decomposition, 4-bit QLoRA running on a single consumer GPU — with complete code, the math behind each approach, and when to choose which.

13 — Go Deeper

📄 Neural Machine Translation of Rare Words with Subword Units (BPE) — Sennrich et al. (2016)
The paper that brought BPE to NLP. Short (8 pages), clear, and still the canonical reference. Section 3 describes the algorithm; Table 1 shows the vocabulary size tradeoffs.

📄 SentencePiece: A simple and language independent subword tokenizer — Kudo & Richardson (2018)
The SentencePiece paper. Section 3 explains the ▁ whitespace handling; Section 4 covers the Unigram language model algorithm. Essential for understanding LLaMA and T5's tokenizers.

📄 Google's Neural Machine Translation System — Schuster & Nakamura (WordPiece)
WordPiece's first public description in the context of Google's production MT system. Section 3.1 covers the PMI-based merge criterion that distinguishes it from BPE.

🔗 HuggingFace Tokenizers Documentation
The fastest production tokenizer library (Rust-backed, Python bindings). The "Quicktour" and "Training from scratch" guides are the best practical starting points for building custom tokenizers.

🔗 Tiktokenizer — interactive GPT tokenizer visualiser
Paste any text and see exactly how GPT-3.5 / GPT-4 tokenizes it, with colour-coded tokens and counts. The fastest way to develop intuition for tokenization costs and patterns.