Positional Encodings:
How Transformers Know Where Words Are

"Dog bites man" and "Man bites dog" are very different events.
To a bag-of-words model, they are identical.
To an attention layer with no positional information — they are also identical.

Attention is extraordinarily good at understanding relationships between words.
But it has no idea which word came first.

Positional encodings are the solution that made the transformer complete.

01 — The Problem: Attention Has No Memory of Order

In Day 24 we built the full attention mechanism. Recall the formula:

Notice what's missing: position. The dot product Q·K^T measures semantic similarity between token representations — it has no term for "how far apart are these tokens?" or "which comes first?" If you give the transformer the sentence "The cat sat on the mat" in shuffled order — "mat the on sat cat the" — the attention scores between "cat" and "sat" are identical in both cases. The model sees the same bag of token embeddings either way.

This is called permutation equivariance: permute the input tokens, and the attention outputs permute in the same way. There is no mechanism inside attention itself that distinguishes position 1 from position 10. For some tasks this doesn't matter much. For language, it is catastrophic — word order is meaning.

The solution is to add a position-dependent signal to each token's embedding before it enters the transformer. The challenge: the signal must be:

02 — Sinusoidal Encodings: The 2017 Original

The original "Attention Is All You Need" paper (Vaswani et al., 2017) introduced sinusoidal positional encodings — a fixed, non-learned formula that assigns every (position, dimension) pair a unique value using sine and cosine waves at different frequencies.

What Does This Actually Do?

Think of it like a clock — but one with many hands. A normal clock has three hands (seconds, minutes, hours), each rotating at a different speed. At any moment, the combination of all three hand positions uniquely identifies the time. No two moments in a 12-hour cycle produce exactly the same combination of three hand positions. Sinusoidal PE works exactly the same way, but with hundreds of "hands" instead of three:

• Low dimensions (small i) → fast waves: these dimensions change rapidly from position to position, like the seconds hand. They help distinguish nearby positions (position 1 vs position 2).
• High dimensions (large i) → slow waves: these dimensions change slowly across positions, like the hours hand. They help distinguish positions that are far apart (position 100 vs position 500).
• The combination of all d_model dimensions gives every position a unique "fingerprint" — like a unique combination of all clock hands. No two positions share the same encoding.

The divisor 10000 is chosen so the lowest-frequency wave has a period of ~62,832 positions — far longer than any sequence seen in 2017. Each pair of dimensions (sin, cos at frequency f) forms a 2D rotation at that frequency, which is precisely why this method works well with dot products.

The Key Mathematical Property

The crucial insight the paper highlights: PE(pos + k) is a linear function of PE(pos). Specifically, there exists a rotation matrix M_k such that PE(pos + k) = M_k · PE(pos). This means relative positions can be expressed as linear transformations — and linear transformations play nicely with the linear projections inside attention. The model can learn to attend to "the token 3 positions back" by learning an appropriate weight matrix, because relative displacement has a consistent linear representation.

Visual Intuition: The Encoding Grid

Here's what sinusoidal encodings look like across 8 positions and 8 dimensions — each cell shows the PE value (blue = negative, orange = positive, intensity = magnitude):

Rows = positions 0–7 · Columns = dimensions 0–7 · High freq (left) → Low freq (right)

03 — Learned Absolute Positions: Simpler and Often Better

BERT (2018) and GPT-2 replaced the fixed sinusoidal formula with a learnable lookup table: a matrix of shape (max_seq_len × d_model) that is trained alongside all other model parameters. Position 0 gets embedding P[0], position 1 gets P[1], and so on. These position embeddings are added to the token embeddings before the first transformer layer — exactly like sinusoidal encodings, but the values are learned from data rather than computed by formula.

In practice, learned positions often match or slightly outperform sinusoidal on standard benchmarks. The model can learn whatever positional signal is most useful for the task — it's not constrained to a fixed mathematical pattern. The cost is small (the positional embedding table is tiny compared to the full model) and the benefit is flexibility.

The critical limitation: hard cutoff at max_seq_len. There is no position embedding for position 513 in BERT — the model has never seen it and has nothing to look up. You cannot extrapolate beyond the training length without retraining or architectural changes. This was acceptable in 2018 when 512 tokens covered most NLP tasks. It becomes a severe constraint when building LLMs that need to handle documents, codebases, or conversations spanning thousands of tokens.

04 — RoPE: Rotary Position Embedding

RoPE (Su et al., 2021) is the dominant positional encoding in modern open-source LLMs — used by LLaMA (all versions), Mistral, Qwen, Falcon, Yi, and most models released since 2023. It elegantly solves the main weakness of both sinusoidal and learned absolute positions: neither directly encodes relative distances in the attention score itself.

The Core Idea

Instead of adding a positional encoding to the token embedding before the attention layer, RoPE modifies the query and key vectors directly — by rotating them by an angle proportional to their position. Think of it like setting the hands of a clock to a specific angle before you hand it to the attention mechanism. If word A is at position 3, you rotate its "meaning vector" by 3°. If word B is at position 7, you rotate it by 7°.

Specifically, for a token at position m, its query vector q is rotated by angle mθ, and for position n, its key vector k is rotated by angle nθ. Because both are rotated before the dot product:

This is the key insight: when you rotate both Q and K before computing their dot product, the result depends only on the difference in their rotation angles — which equals the difference in their positions. The model inherently learns relative positional relationships, not just absolute ones. "The token 3 positions back" has a consistent geometric meaning regardless of where in the sequence you are.

How the Rotation Works

RoPE treats the d-dimensional vector as d/2 pairs of 2D coordinates. Each pair is independently rotated by an angle that depends on both the position and the pair index (different frequencies for different pairs — just like sinusoidal PE):

Why RoPE Handles Long Contexts Better

Learned absolute positions have a hard wall at max_seq_len. Sinusoidal positions degrade because they were designed for fixed sequence lengths. RoPE's rotation angle grows linearly with position — and because the dot product depends only on relative rotation, the model learns to interpret relative distances, which generalise better to longer sequences it hasn't seen. With techniques like YaRN (Yet Another RoPE extensioN) and LongRoPE, models trained with RoPE at 4k context can be extended to 128k+ context by adjusting the rotation frequencies — without full retraining. This is why LLaMA-3.1 supports 128k context without architectural changes.

05 — ALiBi: Attention with Linear Biases

ALiBi (Press et al., 2022) takes a completely different approach. Instead of adding anything to the token embeddings or rotating the Q/K vectors, it adds a penalty directly to the attention scores based on the distance between tokens — before the softmax.

The effect: tokens that are far apart receive a penalty proportional to their distance, making them attend to each other less. The slope m controls how quickly this penalty grows — heads with larger m focus on local context, heads with smaller m can attend globally. No learned parameters, no changes to embedding dimensions, no rotation — just a simple subtraction before softmax.

06 — Head-to-Head Comparison

The practical answer for 2026: if you're training or fine-tuning an LLM from scratch, use RoPE. If you're working with a BERT-style encoder for classification or retrieval, learned absolute positions are fine (context rarely exceeds 512 tokens for these tasks). If you need to extend an existing model's context window cheaply, RoPE + YaRN is the standard approach.

07 — Context Length Extension: YaRN and LongRoPE

One of the most practically important applications of understanding positional encodings is context window extension — taking a model trained at 4k context and extending it to 32k, 128k, or beyond. RoPE makes this possible via frequency adjustment.

The Problem: Frequency Mismatch

A LLaMA model trained at 4096 tokens has seen rotation angles from 0 to 4096·θ_i for each dimension pair. At position 8000 (beyond training), the rotation angle 8000·θ_i for high-frequency dimensions wraps around multiple times — the model has never seen these angle values and the attention pattern breaks down. Low-frequency dimensions (large i, slow rotation) are fine because they've barely rotated even at 8000 tokens. The problem is specifically with high-frequency dimensions getting "confused" at positions beyond training length.

YaRN: Scaling Frequencies

YaRN (Peng et al., 2023) extends RoPE context by scaling down the rotation frequencies — stretching the "clock" so that angles that previously completed one cycle per 4096 tokens now complete one cycle per 32768 tokens. Only a short fine-tuning run (~0.1% of original training compute) on long documents is needed to adapt to the new frequencies. LLaMA-3.1's 128k context uses exactly this approach. The key insight: you don't need to retrain from scratch — you just need the model to see enough long-context examples to adjust to the new frequency scale.

08 — Code: All Four Methods Side by Side

Working implementations of sinusoidal PE, learned absolute PE, RoPE, and ALiBi — all in PyTorch, all producing outputs compatible with the transformer block from Day 24.

import torch, math
import torch.nn as nn

# ─────────────────────────────────────────────────────────────────────────
# 1. Sinusoidal Positional Encoding (Vaswani et al. 2017)
# ─────────────────────────────────────────────────────────────────────────
class SinusoidalPE(nn.Module):
    def __init__(self, d_model, max_len=4096):
        super().__init__()
        pe  = torch.zeros(max_len, d_model)
        pos = torch.arange(0, max_len).unsqueeze(1).float()
        div = torch.exp(torch.arange(0, d_model, 2).float()
                        * -(math.log(10000.0) / d_model))
        pe[:, 0::2] = torch.sin(pos * div)    # even dims → sin
        pe[:, 1::2] = torch.cos(pos * div)    # odd  dims → cos
        self.register_buffer('pe', pe.unsqueeze(0))   # (1, max_len, d_model)

    def forward(self, x):
        return x + self.pe[:, :x.size(1)]   # add PE to token embeddings


# ─────────────────────────────────────────────────────────────────────────
# 2. Learned Absolute Positional Encoding (BERT / GPT-2 style)
# ─────────────────────────────────────────────────────────────────────────
class LearnedPE(nn.Module):
    def __init__(self, d_model, max_len=512):
        super().__init__()
        self.pe = nn.Embedding(max_len, d_model)   # fully learnable lookup table

    def forward(self, x):
        B, T, _ = x.shape
        positions = torch.arange(T, device=x.device)   # [0, 1, 2, ..., T-1]
        return x + self.pe(positions)          # add learned PE to token embeddings


# ─────────────────────────────────────────────────────────────────────────
# 3. RoPE — Rotary Position Embedding (Su et al. 2021)
#    Applied to Q and K vectors INSIDE attention — not to the token embedding
# ─────────────────────────────────────────────────────────────────────────
def precompute_rope_freqs(d_k, max_len=4096, base=10000):
    """Precompute cosine and sine tables for RoPE rotation."""
    theta = 1.0 / (base ** (torch.arange(0, d_k, 2).float() / d_k))
    positions = torch.arange(max_len).float()
    angles = torch.outer(positions, theta)          # (max_len, d_k/2)
    freqs_cos = torch.cos(angles)                   # (max_len, d_k/2)
    freqs_sin = torch.sin(angles)                   # (max_len, d_k/2)
    return freqs_cos, freqs_sin

def apply_rope(x, freqs_cos, freqs_sin):
    """
    Apply RoPE rotation to Q or K vectors.
    x: (batch, heads, seq_len, d_k)
    """
    B, H, T, d_k = x.shape
    cos = freqs_cos[:T].unsqueeze(0).unsqueeze(0)   # (1, 1, T, d_k/2)
    sin = freqs_sin[:T].unsqueeze(0).unsqueeze(0)

    x1 = x[..., :0::2]    # Wait — cleaner to reshape into pairs
    x_r = x.reshape(B, H, T, d_k // 2, 2)         # (B, H, T, d_k/2, 2)
    x_even = x_r[..., 0]                            # (B, H, T, d_k/2)
    x_odd  = x_r[..., 1]

    # Apply 2D rotation: [x_even, x_odd] → [x_even·cos − x_odd·sin, x_even·sin + x_odd·cos]
    rotated_even = x_even * cos - x_odd * sin
    rotated_odd  = x_even * sin + x_odd * cos

    # Interleave back: (B, H, T, d_k/2, 2) → (B, H, T, d_k)
    out = torch.stack([rotated_even, rotated_odd], dim=-1).reshape(B, H, T, d_k)
    return out


# ─────────────────────────────────────────────────────────────────────────
# 4. ALiBi — Attention with Linear Biases (Press et al. 2022)
#    Returns an additive bias matrix to subtract from attention scores
# ─────────────────────────────────────────────────────────────────────────
def build_alibi_bias(n_heads, seq_len):
    """
    Returns (n_heads, seq_len, seq_len) bias matrix.
    Subtract from attention scores before softmax.
    """
    # Head-specific slopes: m_h = 2^(-8h/H) for h = 1 … H
    slopes = torch.pow(2, -torch.arange(1, n_heads + 1).float() * (8 / n_heads))

    # Distance matrix: entry [i,j] = |i − j|
    positions = torch.arange(seq_len)
    distances = (positions.unsqueeze(0) - positions.unsqueeze(1)).abs().float()  # (T, T)

    # Bias = −slope × distance, one per head
    bias = -slopes.view(n_heads, 1, 1) * distances.unsqueeze(0)  # (H, T, T)
    return bias


# ─────────────────────────────────────────────────────────────────────────
# Quick sanity checks
# ─────────────────────────────────────────────────────────────────────────
d_model, seq_len = 64, 8
x = torch.randn(2, seq_len, d_model)   # batch=2

# Sinusoidal
sin_pe = SinusoidalPE(d_model)
out_sin = sin_pe(x)
print(f"Sinusoidal PE output: {out_sin.shape}")
print(f"  PE values differ by position: {not torch.allclose(sin_pe.pe[0,0], sin_pe.pe[0,3])}")

# Learned
learned_pe = LearnedPE(d_model)
out_learned = learned_pe(x)
print(f"Learned PE output:    {out_learned.shape}  ({sum(p.numel() for p in learned_pe.parameters()):,} params)")

# RoPE
cos_f, sin_f = precompute_rope_freqs(d_k=d_model, max_len=128)
q = torch.randn(2, 4, seq_len, d_model)   # (batch, heads, seq, d_k)
q_rotated = apply_rope(q, cos_f, sin_f)
print(f"RoPE Q rotated:       {q_rotated.shape}  (same shape, values rotated)")
print(f"  Rotation changes values: {not torch.allclose(q, q_rotated)}")

# ALiBi
alibi = build_alibi_bias(n_heads=4, seq_len=seq_len)
print(f"ALiBi bias shape:     {alibi.shape}  ← add to attn scores before softmax")
print(f"  Head-0 bias [0→0]: {alibi[0,0,0]:.3f} (same pos, no penalty)")
print(f"  Head-0 bias [0→3]: {alibi[0,0,3]:.3f} (3 apart, penalty applied)")
print(f"  Head-3 bias [0→3]: {alibi[3,0,3]:.3f} (same distance, smaller slope = less penalty)")

09 — Why This Matters in Practice

Your choice of PE determines your context ceiling. BERT's learned positions hard-cap at 512 tokens — trying to run BERT on a 600-token input requires truncation or a different model. GPT-2 caps at 1024. Models using RoPE can be extended post-training via frequency scaling; models using learned absolute positions cannot. When you see a model advertise "128k context" — that capability is almost always a consequence of RoPE + YaRN fine-tuning.

PE affects what the model knows about structure. ALiBi's local-focus bias means models using it naturally pay more attention to nearby context, which can help with tasks where local coherence matters but may limit very long-range reasoning. RoPE's relative position awareness means the model has a consistent notion of "3 tokens back" regardless of absolute position — useful for code (function calls, variable scoping) and long documents where patterns repeat at varying positions.

PE matters for fine-tuning and extending base models. If you fine-tune a LLaMA-3 model (RoPE, 8k base) on longer documents, the model degrades gracefully beyond 8k tokens because RoPE's relative representation is smooth. If you try to fine-tune BERT on longer inputs, you hit a hard wall — you'd need to add new position embeddings and retrain. Understanding PE is essential for any practitioner working on RAG (Day 29) or long-document applications.

10 — A Thought to Sit With

11 — Coming Up Next

Tomorrow — Day 26: BERT vs GPT. We now have the complete transformer with positional encodings. Day 26 splits the same architecture in two directions: the encoder (BERT) that reads the whole sentence at once with bidirectional attention, and the decoder (GPT) that generates one token at a time with causal masking. We cover what each is actually good at, how pre-training objectives differ (masked vs causal language modelling), and why the industry converged on decoder-only models for modern LLMs.

12 — Go Deeper

📄 Attention Is All You Need — Vaswani et al. (2017)
Section 3.5 covers sinusoidal encodings. The choice of 10000 as base and the discussion of linear offset properties is in the appendix.

📄 RoFormer: Enhanced Transformer with Rotary Position Embedding — Su et al. (2021)
The original RoPE paper. Section 3 derives the rotation formulation from scratch. Required reading if you plan to implement or modify RoPE.

📄 Train Short, Test Long: ALiBi — Press et al. (2022)
The ALiBi paper. Table 1 shows the perplexity comparison at lengths beyond training — the extrapolation advantage is striking.

📄 YaRN: Efficient Context Window Extension of Large Language Models
How to extend LLaMA from 4k to 128k context using RoPE frequency scaling. Sections 2–3 are the practical how-to.

▶ Positional Encoding in Transformer — Aladdin Persson
YouTube · 18 min · Clear visual walkthrough of the sinusoidal formula with code. Good complement to this post.