BERT vs GPT:
Encoders, Decoders & What They're For

Two students take the same exam.
One can flip to any page — reading forward, backward, skipping around.
The other can only read left to right, never peeking ahead.

The first student is better at understanding. The second is better at writing.

BERT is the first student. GPT is the second.
Same exam. Same knowledge. Completely different strategies.

01 — One Architecture, Two Families

In Day 24 we built the full transformer architecture. In Day 25 we added positional encodings. The transformer is now complete — but here's something remarkable: BERT and GPT are both transformers. They use the same multi-head attention, the same feed-forward blocks, the same layer norm and residual connections. The difference between them is not the architecture — it's what part of the sequence each token is allowed to see during training.

This distinction flows from a single change in the attention mask. When computing self-attention, the mask determines which positions each token can attend to:

That grid is not just a visual — it's literally the attention mask applied before the softmax in Day 24. BERT sets all mask values to 0 (attend everywhere). GPT sets the upper triangle to −∞ (block future tokens). Everything else — the Q/K/V projections, multi-head attention, FFN layers, residuals — is identical.

02 — BERT: The Encoder

What BERT Is

BERT (Bidirectional Encoder Representations from Transformers — Google, 2018) is an encoder-only transformer. It takes a sequence of tokens and produces a rich contextual embedding for each token — a representation that captures the meaning of that token in the context of the entire sentence, both what comes before and what comes after.

BERT's Pre-Training: Masked Language Modelling (MLM)

BERT was pre-trained on two tasks simultaneously. The first and most important is Masked Language Modelling (MLM): randomly hide 15% of tokens in the input by replacing them with a special [MASK] token, then train the model to predict what the hidden tokens were.

To predict "sat," the model must understand both the subject ("The cat") and the destination ("on the [MASK]"). It cannot rely on just looking left — it must use both sides. This forces BERT to build genuinely bidirectional representations. After pre-training on 3.3 billion words, every layer of BERT learned something deep about language: syntax, coreference, semantic relationships, named entities — all encoded in the token embeddings.

The second task is Next Sentence Prediction (NSP): given two sentences A and B, predict whether B actually follows A in the original document or was randomly sampled. This teaches BERT cross-sentence relationships — important for tasks like question answering where the question and passage are both inputs.

The [CLS] Token — BERT's Summary Vector

BERT prepends a special [CLS] (classification) token to every input. After passing through all transformer layers, the final embedding of this token aggregates information from the entire sequence — it becomes a sentence-level representation. For classification tasks, you attach a small linear head on top of [CLS] and fine-tune. The rest of BERT's weights don't need to change much — they already understand language remarkably well.

BERT Fine-Tuning: One Pre-trained Model, Any Task

The pre-train/fine-tune paradigm BERT introduced was revolutionary: train once on massive unlabelled text, then fine-tune on small labelled datasets for any specific task. A single BERT-base model reached state-of-the-art on 11 NLP benchmarks simultaneously in 2018 — sentiment analysis, named entity recognition, question answering, textual entailment, and more. The key insight: language understanding is universal. A model that deeply understands language in general can quickly learn any language task with a small additional training signal.

03 — GPT: The Decoder

What GPT Is

GPT (Generative Pre-trained Transformer — OpenAI, 2018) is a decoder-only transformer. It takes a sequence of tokens and, one token at a time, predicts the next one. The causal mask ensures that when predicting token at position n, it can only see tokens 1 through n−1 — never anything after. This constraint is what makes generation possible.

GPT's Pre-Training: Causal Language Modelling (CLM)

GPT's training objective is beautifully simple: predict the next token. Given a sequence of tokens, maximise the probability of each token given all preceding tokens. No masking tricks, no special tokens — just next-token prediction across the entire training corpus.

This objective is strikingly powerful because it provides a training signal at every position in every sequence. A 1,024-token sequence generates 1,024 training examples in one forward pass. And the task is hard enough to force the model to learn everything about language: grammar (wrong grammar is less probable), facts (wrong facts are less probable), logic (incoherent continuations are less probable), and style — all emerge as side effects of trying to predict what word comes next.

Generation: How GPT Produces Text

At inference time, GPT generates text autoregressively — one token at a time, feeding each newly generated token back as input for the next step:

The sampling strategy matters enormously here. Greedy decoding (always pick the highest probability token) produces repetitive, boring text. Temperature scaling (divide logits by T before softmax) controls randomness — low T makes text more deterministic, high T more creative/random. Top-k sampling (only sample from the top-k tokens) and nucleus sampling / top-p (sample from the smallest set of tokens whose cumulative probability exceeds p) are used by GPT-3, GPT-4, and most modern LLMs.

04 — The Attention Mask: The One Real Difference

Everything above comes down to a single change in the attention computation. Here is the exact mask difference:

This one mask is the entire architectural difference between BERT and GPT. Both use:

• The same transformer blocks (attention + FFN + residual + LayerNorm)
• The same token embeddings and positional encodings
• The same training procedure (AdamW optimiser, learning rate warmup)
• The same tokenizer format (WordPiece for BERT, BPE for GPT)

The mask determines the pre-training objective — bidirectional attention enables MLM, causal attention enables CLM — and the pre-training objective determines what the model is good at.

05 — BERT vs GPT: Head-to-Head

06 — The Third Family: Encoder-Decoder (T5, BART)

There's a third architecture that combines both: the encoder-decoder (or seq2seq) transformer. It uses a BERT-style encoder to read and understand the input, and a GPT-style decoder to generate the output. The decoder can attend to all encoder outputs via cross-attention — a third type of attention head that reads from the encoder's representations while generating.

07 — Why the Industry Converged on Decoder-Only

By 2023 the industry had converged overwhelmingly on decoder-only models for general-purpose LLMs — GPT-4, Claude, LLaMA, Mistral, Gemma, Falcon, Qwen. BERT-family models are still dominant for embedding and retrieval. But for the "chat with an AI" use case, decoder-only won. Here's why:

08 — Notable Models in Each Family

Encoder-Only — The BERT Family

Decoder-Only — The GPT Family

Encoder-Decoder — The T5 Family

09 — Practical Decision Guide: Which to Use?

10 — Code: Both Architectures Side by Side

A minimal BERT-style encoder and GPT-style decoder in PyTorch, showing the exact difference in the attention mask.

import torch
import torch.nn as nn
import torch.nn.functional as F
import math

# ─────────────────────────────────────────────────────────────────────────
# Shared transformer block (identical for both BERT and GPT)
# ─────────────────────────────────────────────────────────────────────────
class TransformerBlock(nn.Module):
    def __init__(self, d_model=128, n_heads=4, d_ff=512, dropout=0.1):
        super().__init__()
        self.attn  = nn.MultiheadAttention(d_model, n_heads, dropout=dropout, batch_first=True)
        self.ff    = nn.Sequential(nn.Linear(d_model, d_ff), nn.GELU(), nn.Linear(d_ff, d_model))
        self.norm1 = nn.LayerNorm(d_model)
        self.norm2 = nn.LayerNorm(d_model)
        self.drop  = nn.Dropout(dropout)

    def forward(self, x, attn_mask=None):
        # Pre-LN + residual (modern stable training variant)
        x_norm = self.norm1(x)
        attn_out, _ = self.attn(x_norm, x_norm, x_norm, attn_mask=attn_mask)
        x = x + self.drop(attn_out)
        x = x + self.drop(self.ff(self.norm2(x)))
        return x


# ─────────────────────────────────────────────────────────────────────────
# BERT-style encoder — bidirectional, no mask
# ─────────────────────────────────────────────────────────────────────────
class BERTEncoder(nn.Module):
    def __init__(self, vocab_size=30522, d_model=128, n_layers=4,
                 n_heads=4, max_len=512):
        super().__init__()
        self.tok_emb = nn.Embedding(vocab_size, d_model)
        self.pos_emb = nn.Embedding(max_len, d_model)  # learned absolute PE (BERT-style)
        self.layers  = nn.ModuleList([TransformerBlock(d_model, n_heads) for _ in range(n_layers)])
        self.norm    = nn.LayerNorm(d_model)

    def forward(self, token_ids):
        B, T = token_ids.shape
        positions = torch.arange(T, device=token_ids.device)
        x = self.tok_emb(token_ids) + self.pos_emb(positions)
        for layer in self.layers:
            x = layer(x, attn_mask=None)   # ← NO MASK: full bidirectional attention
        return self.norm(x)   # (B, T, d_model) — all token representations


# ─────────────────────────────────────────────────────────────────────────
# GPT-style decoder — causal (left-to-right) mask
# ─────────────────────────────────────────────────────────────────────────
class GPTDecoder(nn.Module):
    def __init__(self, vocab_size=50257, d_model=128, n_layers=4,
                 n_heads=4, max_len=1024):
        super().__init__()
        self.tok_emb = nn.Embedding(vocab_size, d_model)
        self.pos_emb = nn.Embedding(max_len, d_model)
        self.layers  = nn.ModuleList([TransformerBlock(d_model, n_heads) for _ in range(n_layers)])
        self.norm    = nn.LayerNorm(d_model)
        self.lm_head = nn.Linear(d_model, vocab_size, bias=False)  # predicts next token

    def get_causal_mask(self, T, device):
        # Upper triangle = -inf: token at position i cannot see positions j > i
        return torch.triu(torch.full((T, T), float('-inf'), device=device), diagonal=1)

    def forward(self, token_ids):
        B, T = token_ids.shape
        positions = torch.arange(T, device=token_ids.device)
        x = self.tok_emb(token_ids) + self.pos_emb(positions)
        mask = self.get_causal_mask(T, token_ids.device)
        for layer in self.layers:
            x = layer(x, attn_mask=mask)  # ← CAUSAL MASK: only attend left
        x = self.norm(x)
        return self.lm_head(x)   # (B, T, vocab_size) — next-token logits

    @torch.no_grad()
    def generate(self, prompt_ids, max_new_tokens=20, temperature=1.0):
        """Simple autoregressive generation with temperature sampling."""
        ids = prompt_ids.clone()
        for _ in range(max_new_tokens):
            logits = self(ids)              # (1, T, vocab)
            next_logits = logits[:, -1, :] / temperature   # last position
            probs = F.softmax(next_logits, dim=-1)
            next_id = torch.multinomial(probs, num_samples=1)
            ids = torch.cat([ids, next_id], dim=1)          # append and loop
        return ids


# ─────────────────────────────────────────────────────────────────────────
# BERT classification head (add on top for fine-tuning)
# ─────────────────────────────────────────────────────────────────────────
class BERTClassifier(nn.Module):
    def __init__(self, encoder, num_classes=2, d_model=128):
        super().__init__()
        self.encoder = encoder
        self.head = nn.Linear(d_model, num_classes)

    def forward(self, token_ids):
        out = self.encoder(token_ids)     # (B, T, d_model)
        cls_repr = out[:, 0, :]           # [CLS] token is at position 0
        return self.head(cls_repr)         # (B, num_classes) — classification logits


# ─────────────────────────────────────────────────────────────────────────
# Sanity checks
# ─────────────────────────────────────────────────────────────────────────
B, T = 2, 16   # batch=2, sequence_length=16
fake_input = torch.randint(0, 1000, (B, T))

bert = BERTEncoder(vocab_size=1000, d_model=128)
gpt  = GPTDecoder(vocab_size=1000, d_model=128)
clf  = BERTClassifier(bert, num_classes=3)

bert_out   = bert(fake_input)
gpt_logits = gpt(fake_input)
clf_out    = clf(fake_input)

# Check causal mask: position 0 should only attend to position 0
mask = gpt.get_causal_mask(4, 'cpu')

print(f"BERT output:         {bert_out.shape}   ← (batch, seq, d_model)")
print(f"GPT logits:          {gpt_logits.shape}  ← (batch, seq, vocab_size)")
print(f"Classifier output:   {clf_out.shape}    ← (batch, num_classes)")
print(f"\nCausal mask (4×4):\n{mask}")
print(f"\nBERT params:  {sum(p.numel() for p in bert.parameters()):,}")
print(f"GPT params:   {sum(p.numel() for p in gpt.parameters()):,}")

11 — A Thought to Sit With

12 — Coming Up Next

Tomorrow — Day 27: Tokenization Deep Dive. We've been using tokenizers throughout the series without fully opening the black box. Day 27 goes inside: Byte-Pair Encoding (BPE) — how GPT builds its vocabulary; WordPiece — how BERT does it differently; SentencePiece — the language-agnostic approach used by LLaMA and T5; and why the same word tokenizes into completely different numbers of tokens across models — with direct consequences for cost, context length, and multilingual performance.

13 — Go Deeper

📄 BERT: Pre-training of Deep Bidirectional Transformers — Devlin et al. (2018)
The original BERT paper. Section 3 (model architecture) and Section 4 (fine-tuning) are the essential parts. Table 1 showing 11 GLUE benchmark improvements simultaneously is one of the most impressive result tables in NLP history.

📄 Improving Language Understanding by Generative Pre-Training — Radford et al. (2018)
The original GPT-1 paper. Short and readable. Section 3 explains the pre-training/fine-tuning paradigm that the entire field adopted.

📄 RoBERTa: A Robustly Optimized BERT Pretraining Approach — Liu et al. (2019)
Shows that BERT was undertrained. Removing NSP, increasing batch size, and training longer produces consistent improvements. Section 4 is a clean ablation study worth reading.

📄 Exploring the Limits of Transfer Learning with T5 — Raffel et al. (2019)
Unifies every NLP task as text-to-text. The most rigorous comparison of architectures, objectives, and scales ever published. Section 3 is a masterclass in ablation study design.

▶ BERT explained — Andrej Karpathy / various
Karpathy's "Let's build GPT from scratch" on YouTube is the clearest derivation of the decoder architecture available. Highly recommended alongside this post.