Attention & the Transformer:
The Architecture That Changed Everything

"The animal didn't cross the street because it was too tired."
What does "it" refer to — the animal, or the street?
You knew instantly. You attended to the right word.

For a decade, building a machine that could do the same
was one of the hardest problems in AI.
In 2017, eight engineers solved it with a single scaled dot product.

01 — The Central Idea: Every Word Talks to Every Other Word

Before transformers, language models processed text like reading a book through a keyhole — one word at a time, left to right. To connect "it" to "animal" in a 50-word sentence, information had to travel through 48 hidden state updates, each one potentially overwriting what came before. The further apart two words were, the harder it was to connect them.

The transformer throws this constraint away entirely. Its core idea is strikingly simple: let every word look at every other word at the same time, and let the model decide how much each pair is related. Not "word 1 then word 2 then word 3" — all words simultaneously, all relationships computed in parallel.

02 — The Core Question: How Does a Word Decide What to Pay Attention To?

Here is the problem the architects had to solve. You have a sentence: "The cat sat on the mat because it was comfortable." The word "it" needs to figure out what it refers to. It can look at every other word — but how does it score them? How does it decide "mat" is more relevant than "sat" here?

The answer the paper proposed is elegant: split each word's representation into three different roles, and use those roles to compute relationships. Every word simultaneously acts as:

1. Something that is looking for relevant information from others
2. Something that announces what it can offer to those looking
3. Something that delivers the actual information once it is selected

These three roles are named Query, Key, and Value — borrowed from the world of databases and information retrieval. Understanding what each one truly means is the key to understanding the entire transformer.

03 — Q, K, V: What They Are and Why They're Named That Way

Why Are Key and Value Two Separate Things?

This is the question most explanations skip — and it is the most important one. Why not just match directly on the word embedding, without the Key/Value separation?

The answer is flexibility and context-sensitivity. A word needs to advertise itself differently depending on the relationship type being computed, and deliver different content depending on who is asking. The Key and Value are both learned projections of the same word — but through different weight matrices, which means the model learns to separate "how I want to be found" from "what I give when found."

The Three Projections — The Learned Matrices

Every word in the sequence produces all three vectors through three separate learned linear transformations (W_Q, W_K, W_V) — weight matrices trained during pre-training:

04 — The Full Attention Computation: Step by Step

Now that Q, K, and V make intuitive sense, let us trace the full computation. Four steps, each with a clear purpose.

Step 1 — Raw Scores: How Well Does Each Query Match Each Key?

Dot product every token's Query against every other token's Key. The higher the dot product, the more aligned the two vectors are — the more relevant that word is to the query.

Step 2 — Scale: Prevent Softmax Saturation

Step 3 — Softmax: Convert Scores to Attention Weights

Step 4 — Weighted Sum of Values: Collect the Content

All four steps in one formula — the heart of every transformer ever built:

05 — Worked Numeric Example: Seeing It Happen

Full computation for a 3-word sequence "Cat sat mat", d_k = 2. The Q, K, V values below are what the model produces after projecting through W_Q, W_K, W_V.

Step 1: Raw scores for "Cat" (row 0 of Q · Kᵀ)

Cat→Cat: (1.0×1.0)+(0.5×0.2) = 1.10 ← Cat querying itself
Cat→sat: (1.0×0.5)+(0.5×1.0) = 1.00 ← Cat querying "sat"
Cat→mat: (1.0×0.2)+(0.5×0.8) = 0.60 ← Cat querying "mat"

Step 2: Scale by 1/√2 ≈ 0.707

[1.10, 1.00, 0.60] × 0.707 = [0.778, 0.707, 0.424]

Step 3: Softmax → attention weights

exp([0.778, 0.707, 0.424]) = [2.177, 2.028, 1.528]  sum = 5.733
weights = [0.380, 0.354, 0.266]
↑ "Cat" pays 38% to itself, 35.4% to "sat", 26.6% to "mat"

Step 4: Weighted sum of Values

0.380 × [2.0, 0.0] = [0.760, 0.000] ← from Cat's Value
0.354 × [0.0, 2.0] = [0.000, 0.708] ← from sat's Value
0.266 × [1.0, 1.0] = [0.266, 0.266] ← from mat's Value
─────────────────────────────────────
"Cat" new representation = [1.026, 0.974]

06 — Multi-Head Attention: Attending to Many Relationships at Once

Single-head attention produces one score per token pair — one relationship type. But language has many simultaneous relationship types: syntactic (subject-verb agreement), semantic (synonymy), coreference (pronoun resolution), positional (nearby context). A single head must average over all of them. It is like trying to answer multiple questions from a group at once — you miss the nuance.

Multi-head attention runs H independent attention operations in parallel, each with its own W_Q, W_K, W_V. Each head learns to specialise in a different type of relationship.

07 — Positional Encodings: Adding the Sense of Order

Attention is position-blind. The dot product Q·Kᵀ measures semantic similarity — no notion of which word came first. "Cat sat mat" and "Mat sat cat" look identical to attention alone. Positional encodings add a position-dependent signal to each word's embedding before attention runs, breaking this symmetry.

Day 25 covers all four in full depth — the math, the intuition, and why RoPE became the dominant choice for every modern LLM.

08 — Causal Masking: How GPT Stays Honest During Training

Standard attention lets every word see every other word — including future ones. For a model learning to predict the next word, seeing future words is cheating: you can just copy the answer. Decoder models (GPT, LLaMA, Claude) apply a causal mask: add −∞ to future positions before softmax. Since exp(−∞) = 0, future tokens get exactly zero weight.

Causal mask for seq_len=4:

      tok1   tok2   tok3   tok4
tok1 [ 0.0, -inf, -inf, -inf] ← sees only itself
tok2 [ 0.0,  0.0, -inf, -inf] ← sees tok1 and itself
tok3 [ 0.0,  0.0,  0.0, -inf] ← sees tok1, tok2, itself
tok4 [ 0.0,  0.0,  0.0,  0.0] ← sees all previous

At inference: the model reads "The capital of France is" and predicts "Paris." It appends "Paris" and predicts the next token. One token at a time, left to right — exactly matching how it was trained.

09 — The Complete Transformer Block

A transformer stacks identical blocks. Each block has two sub-layers, each wrapped with a residual connection and layer normalisation:

10 — Code: Attention and Full Transformer Block from Scratch

import torch, math
import torch.nn as nn
import torch.nn.functional as F

# ─────────────────────────────────────────────────────────────────────────
# 1. Scaled Dot-Product Attention
#    Q: what I'm looking for  |  K: what I can offer  |  V: what I deliver
# ─────────────────────────────────────────────────────────────────────────
def scaled_dot_product_attention(Q, K, V, mask=None):
    d_k = Q.size(-1)
    scores = (Q @ K.transpose(-2, -1)) / math.sqrt(d_k)   # (B, H, T, T)
    if mask is not None:
        scores = scores + mask
    weights = F.softmax(scores, dim=-1)
    return weights @ V, weights


# ─────────────────────────────────────────────────────────────────────────
# 2. Multi-Head Attention — H heads, each specialising in different relationships
# ─────────────────────────────────────────────────────────────────────────
class MultiHeadAttention(nn.Module):
    def __init__(self, d_model, n_heads):
        super().__init__()
        assert d_model % n_heads == 0
        self.d_k     = d_model // n_heads
        self.n_heads = n_heads
        self.W_q = nn.Linear(d_model, d_model, bias=False)  # what am I looking for?
        self.W_k = nn.Linear(d_model, d_model, bias=False)  # what can I offer?
        self.W_v = nn.Linear(d_model, d_model, bias=False)  # what do I deliver?
        self.W_o = nn.Linear(d_model, d_model, bias=False)  # mix all heads' outputs

    def forward(self, x, mask=None):
        B, T, C = x.shape
        def split(t):
            return t.view(B, T, self.n_heads, self.d_k).transpose(1, 2)
        Q, K, V = split(self.W_q(x)), split(self.W_k(x)), split(self.W_v(x))
        out, attn = scaled_dot_product_attention(Q, K, V, mask)
        out = out.transpose(1, 2).contiguous().view(B, T, C)
        return self.W_o(out), attn


# ─────────────────────────────────────────────────────────────────────────
# 3. Feed-Forward Network (position-wise, 4× expansion, GELU)
# ─────────────────────────────────────────────────────────────────────────
class FeedForward(nn.Module):
    def __init__(self, d_model):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(d_model, d_model * 4),
            nn.GELU(),
            nn.Linear(d_model * 4, d_model),
        )
    def forward(self, x): return self.net(x)


# ─────────────────────────────────────────────────────────────────────────
# 4. Transformer Block (Pre-LN variant — more stable than original Post-LN)
# ─────────────────────────────────────────────────────────────────────────
class TransformerBlock(nn.Module):
    def __init__(self, d_model, n_heads, dropout=0.1):
        super().__init__()
        self.ln1  = nn.LayerNorm(d_model)
        self.attn = MultiHeadAttention(d_model, n_heads)
        self.ln2  = nn.LayerNorm(d_model)
        self.ffn  = FeedForward(d_model)
        self.drop = nn.Dropout(dropout)

    def forward(self, x, mask=None):
        attn_out, w = self.attn(self.ln1(x), mask)
        x = x + self.drop(attn_out)           # residual
        x = x + self.drop(self.ffn(self.ln2(x)))
        return x, w


# ─────────────────────────────────────────────────────────────────────────
# 5. Sinusoidal Positional Encoding
# ─────────────────────────────────────────────────────────────────────────
class SinusoidalPE(nn.Module):
    def __init__(self, d_model, max_len=4096):
        super().__init__()
        pe  = torch.zeros(max_len, d_model)
        pos = torch.arange(0, max_len).unsqueeze(1).float()
        div = torch.exp(torch.arange(0, d_model, 2).float() * -(math.log(10000.0) / d_model))
        pe[:, 0::2] = torch.sin(pos * div)
        pe[:, 1::2] = torch.cos(pos * div)
        self.register_buffer('pe', pe.unsqueeze(0))
    def forward(self, x): return x + self.pe[:, :x.size(1)]


# ─────────────────────────────────────────────────────────────────────────
# Sanity check
# ─────────────────────────────────────────────────────────────────────────
d_model, n_heads, seq_len = 128, 4, 16
pe    = SinusoidalPE(d_model)
block = TransformerBlock(d_model, n_heads)
x     = pe(torch.randn(2, seq_len, d_model))
mask  = torch.triu(torch.full((seq_len, seq_len), float('-inf')), diagonal=1)
out, attn_w = block(x, mask=mask)
print(f"Input  shape:       {x.shape}")
print(f"Output shape:       {out.shape}")
print(f"Attn weight shape:  {attn_w.shape}")
print(f"Params in block:    {sum(p.numel() for p in block.parameters()):,}")
print(f"Causal mask OK:     {(attn_w[0, 0, 0, 1:] == 0).all().item()}")
print(f"Attn weights sum:   {attn_w[0, 0, 0].sum().item():.4f}")

11 — KV-Cache: Why Chat Generation Is Fast

During text generation, the model produces one token at a time. Without optimisation, it would recompute K and V for every token in context at every step. The KV-cache stores K and V once computed and reuses them. Only the new token's K and V need to be computed each step.

This reduces generation complexity from O(n²) (recompute everything) to O(n) (compute one row, reuse all others). For a 128k-context conversation, the difference is seconds vs minutes per response.

Why cache K and V, but not Q? Because K and V represent fixed properties of each context token — "what I can offer" and "what I deliver" don't change once a token is in the sequence. Q represents what the current generation step is looking for — it changes at every step as the model produces the next token.

12 — Encoder vs Decoder vs Encoder-Decoder

13 — A Thought to Sit With

14 — Coming Up Next

Tomorrow — Day 25: Positional Encodings. Attention is permutation-blind by design — shuffle the tokens and you get the same result. Positional encodings fix this: sinusoidal patterns, learned absolute positions, and the rotary embeddings (RoPE) used in LLaMA, Mistral, and Gemma that encode position as a vector rotation — naturally extensible to sequences far longer than training.

15 — Go Deeper

▶ But what is a GPT? Visual intro to Transformers — 3Blue1Brown
YouTube · 27 min · The best visual explanation of attention available. The Q/K/V segment is particularly clear. Watch alongside this post.

🔗 The Illustrated Transformer — Jay Alammar
Step-by-step visual walkthrough. The multi-head attention and encoder-decoder diagrams are the clearest available.

📄 Attention Is All You Need — Vaswani et al. (2017)
The original. Section 3.2 and Figure 2 are the core. The ablation table (Section 7) shows how each architectural choice was validated.

📄 FlashAttention — Dao et al. (2022)
The engineering that made long-context LLMs practical. Section 2 explains the IO-aware tiling. Essential for LLM inference work.

🔗 A Mathematical Framework for Transformer Circuits — Anthropic (2021)
Foundational mechanistic interpretability work — what do Q, K, V actually compute in a trained model? The heads-as-specialists findings come from here.