Recurrent Neural Networks:
The First Serious Attempt at Sequence Modelling

A feedforward network reads a sentence the way a bad student reads a dictionary —
every word in isolation, with no memory of what came before.

An RNN reads it the way a person reads a novel —
each word understood in light of everything already read.

That was the idea. The execution, as we'll see, was flawed in a beautiful way.

01 — Why Feedforward Networks Cannot Handle Sequences

Everything we've built so far — perceptrons, CNNs, basic deep networks — shares one fundamental assumption: inputs are independent of each other. Feed in an image, get a classification. Feed in another image, get another classification. The model has no memory of the first image when it processes the second.

Language violates this assumption completely. The meaning of every word in a sentence depends on the words that came before it. Consider:

• "The bank is steep." → bank = riverbank
• "The bank is closed." → bank = financial institution
• "I went to the bank after the bank." → two different banks in one sentence

A feedforward network processing "bank" as an isolated token cannot know which meaning applies. It would need to see the entire context — but fixed-size inputs mean you must decide in advance how many words of context to include. A 10-word window misses long-range dependencies. A 500-word window is computationally absurd and still fixed. And crucially, the same pattern (subject → verb → object) means something different at position 1 vs position 50 in a sentence.

What's needed is a model that:

The Recurrent Neural Network was designed to satisfy all four requirements. In 1986, Rumelhart, Hinton and Williams formalised the idea; through the 1990s and 2000s it became the dominant architecture for any task involving sequences: language, speech, music, time series. Let's build it from scratch.

02 — The RNN Cell: Architecture and Forward Pass

An RNN processes a sequence one step at a time. At each step t, it receives two inputs: the current token x_t and a hidden state h_{t-1} carrying all information from previous steps. It combines them to produce a new hidden state h_t, which becomes the memory passed to the next step.

The output at each step can be computed as: y_t = W_y · h_t + b_y. Depending on the task, you might use the output at every step (sequence labelling), only the last step (classification), or generate a new sequence (translation).

The Unrolled View

An RNN is often shown as a loop — one cell that repeatedly applies the same operation. But to understand training, it helps to "unroll" it — draw out each timestep as a separate copy of the cell, all sharing the same weights:

This unrolled view reveals something important: the RNN is essentially a very deep feedforward network where depth = sequence length, and all layers share the same weights. A 100-word sentence is equivalent to a 100-layer network. This is the source of both the RNN's power and its fundamental problem.

Worked Forward Pass

Let's trace through a tiny example manually. Sentence: "I love NLP". hidden_size=2, input_size=2. All weights initialised to 0.5, bias=0, h_0=[0,0].

Step 1: x_1 = "I" = [1, 0]
pre_act = W_h·h_0 + W_x·x_1 = [0,0] + [0.5×1 + 0.5×0, 0.5×1 + 0.5×0] = [0.5, 0.5]
h_1 = tanh([0.5, 0.5]) = [0.462, 0.462]

Step 2: x_2 = "love" = [0, 1]
pre_act = W_h·h_1 + W_x·x_2
        = [0.5×0.462+0.5×0.462, ...] + [0.5×0+0.5×1, ...]
        = [0.462, 0.462] + [0.5, 0.5] = [0.962, 0.962]
h_2 = tanh([0.962, 0.962]) = [0.745, 0.745]

Step 3: x_3 = "NLP" = [1, 1]
pre_act = [0.5×0.745×2, ...] + [0.5×1+0.5×1, ...]
        = [0.745, 0.745] + [1.0, 1.0] = [1.745, 1.745]
h_3 = tanh([1.745, 1.745]) = [0.942, 0.942]  ← final hidden state encodes full sentence

03 — RNN Variants: Many-to-One, One-to-Many, Many-to-Many

The RNN architecture is flexible — by choosing which outputs to use, you can handle a wide range of sequence tasks:

The seq2seq architecture (Sutskever et al., 2014) was the major breakthrough that brought RNNs into production NLP. An encoder RNN reads the source sentence and compresses it into a single fixed-size context vector (the final hidden state). A decoder RNN then uses this context vector to generate the target sentence one token at a time. Google Translate ran on this architecture from 2016–2017.

04 — Backpropagation Through Time: Why Training RNNs Is Hard

Training an RNN means computing gradients and updating the weights W_h, W_x, b. We use backpropagation — but on the unrolled network, gradients flow backwards through time: from the loss at the final step, back through every hidden state, all the way to step 1. This is called Backpropagation Through Time (BPTT).

Let's trace what happens to the gradient as it flows backwards through T steps:

The gradient at step 1 involves multiplying W_h by itself T−1 times and multiplying T−1 tanh derivative terms. This is where the problem emerges — and it splits into two distinct failure modes:

Failure Mode 1: Vanishing Gradients

The tanh derivative sech²(x) = 1 − tanh²(x) has a maximum value of 1.0 (at x=0) and shrinks toward 0 as |x| grows. In practice, during training, the pre-activations of tanh rarely sit exactly at 0 — they spread out and the derivatives are typically well below 1.0. If the largest singular value of W_h is also less than 1 (which is common with standard initialisation), then each multiplication by W_h shrinks the gradient further. Over T steps:

By step T−50, the gradient signal is effectively zero. The weights W_x and W_h at early timesteps receive no meaningful update signal — the model simply cannot learn that "The animal at the beginning of the sentence" is the referent for "it" 50 words later. Long-range dependencies are invisible to the gradient.

Failure Mode 2: Exploding Gradients

The opposite problem: if the largest singular value of W_h is greater than 1, the gradient explodes exponentially over T steps. A gradient that starts at magnitude 1.0 can reach 10^15 after 50 steps with ||W_h|| = 1.5. This causes weight updates so large that the model parameters become NaN. Exploding gradients are the more visible failure — training simply diverges. Vanishing gradients are more insidious — training appears to proceed normally, but the model quietly fails to learn long-range patterns.

05 — What RNNs Are Good At (and Where They Still Win)

It's easy to read the vanishing gradient problem and conclude RNNs are useless. That's wrong. RNNs are excellent for tasks where:

RNNs are also O(1) in memory per step — the hidden state is a fixed-size vector regardless of how many tokens you've processed. A transformer's attention is O(n²) in sequence length. For extremely long sequences where you need constant memory, RNN-style architectures (and their successors like Mamba) are still competitive.

06 — Partial Fixes: What People Tried Before LSTM

The deep learning community didn't give up on RNNs when the vanishing gradient problem was identified. Several partial solutions emerged:

None of these were satisfying. The community needed an architectural solution — a way to give the network a direct gradient path from later timesteps to earlier ones, bypassing the multiplicative chain that caused the decay. That solution was the LSTM, which we'll cover in full in Day 21.

07 — Code: RNN from Scratch + PyTorch Comparison

Three implementations: a manual RNN cell showing every weight matrix, a full sequence processor with gradient norm tracking to visualise vanishing, and the PyTorch built-in for comparison. Then a demonstration of gradient clipping.

import torch
import torch.nn as nn
import numpy as np

# ─────────────────────────────────────────────────────────────────────────
# 1. RNN Cell from scratch — every operation explicit
# ─────────────────────────────────────────────────────────────────────────
class RNNCellManual(nn.Module):
    def __init__(self, input_size, hidden_size):
        super().__init__()
        self.hidden_size = hidden_size
        # Two weight matrices + one bias
        self.W_x = nn.Linear(input_size,  hidden_size, bias=False)  # input → hidden
        self.W_h = nn.Linear(hidden_size, hidden_size, bias=False)  # hidden → hidden (recurrent)
        self.b   = nn.Parameter(torch.zeros(hidden_size))

    def forward(self, x_t, h_prev):
        """
        x_t:    (batch, input_size)
        h_prev: (batch, hidden_size)
        returns h_t: (batch, hidden_size)
        """
        pre_act = self.W_x(x_t) + self.W_h(h_prev) + self.b
        h_t = torch.tanh(pre_act)
        return h_t


# ─────────────────────────────────────────────────────────────────────────
# 2. Full sequence RNN — processes a whole sequence, returns all hidden states
# ─────────────────────────────────────────────────────────────────────────
class SimpleRNN(nn.Module):
    def __init__(self, input_size, hidden_size, output_size):
        super().__init__()
        self.cell    = RNNCellManual(input_size, hidden_size)
        self.fc_out  = nn.Linear(hidden_size, output_size)
        self.hidden_size = hidden_size

    def forward(self, x):
        """x: (batch, seq_len, input_size)"""
        B, T, _ = x.shape
        h = torch.zeros(B, self.hidden_size)   # h_0 initialised to zeros
        hidden_states = []
        for t in range(T):
            h = self.cell(x[:, t, :], h)       # step through sequence
            hidden_states.append(h)
        all_h = torch.stack(hidden_states, dim=1)  # (B, T, hidden_size)
        out   = self.fc_out(all_h[:, -1, :])    # use final hidden state for classification
        return out, all_h


# ─────────────────────────────────────────────────────────────────────────
# 3. Vanishing gradient demonstration
#    Train on a task that requires remembering the FIRST token after T steps
# ─────────────────────────────────────────────────────────────────────────
def demonstrate_vanishing_gradient(seq_lengths=[5, 20, 50, 100]):
    print("Gradient magnitude at step 1 (should be large to learn long-range deps):")
    print(f"{'Seq length':>12} | {'||∂L/∂h_1||':>14} | {'Assessment':>20}")
    print("-" * 52)

    for T in seq_lengths:
        model = SimpleRNN(input_size=8, hidden_size=16, output_size=2)
        x = torch.randn(4, T, 8)           # random sequence
        y = torch.randint(0, 2, (4,))        # random labels

        out, all_h = model(x)
        loss = nn.CrossEntropyLoss()(out, y)
        loss.backward()

        # Gradient of loss w.r.t. the hidden state at step 1 (earliest step)
        grad_h1 = all_h.grad
        if grad_h1 is not None:
            g_norm = grad_h1[:, 0, :].norm().item()
        else:
            # Recompute with retain_grad
            model.zero_grad()
            x2 = torch.randn(4, T, 8, requires_grad=True)
            out2, all_h2 = model(x2)
            all_h2.retain_grad()
            loss2 = nn.CrossEntropyLoss()(out2, y)
            loss2.backward()
            g_norm = all_h2.grad[:, 0, :].norm().item() if all_h2.grad is not None else 0.0

        assessment = "✓ Learnable" if g_norm > 0.01 else ("⚠ Weak" if g_norm > 0.001 else "✗ Vanished")
        print(f"{T:>12} | {g_norm:>14.6f} | {assessment:>20}")


# ─────────────────────────────────────────────────────────────────────────
# 4. Gradient clipping — the standard fix for exploding gradients
# ─────────────────────────────────────────────────────────────────────────
def training_step_with_clipping(model, x, y, optimizer, clip_norm=1.0):
    optimizer.zero_grad()
    out, _ = model(x)
    loss = nn.CrossEntropyLoss()(out, y)
    loss.backward()

    # Check norm BEFORE clipping
    total_norm_before = torch.nn.utils.clip_grad_norm_(model.parameters(), float('inf'))

    # Clip — scale gradient if norm > clip_norm
    torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=clip_norm)

    # Check norm AFTER clipping
    total_norm_after = sum(p.grad.norm()**2 for p in model.parameters() if p.grad is not None)**0.5

    optimizer.step()
    return loss.item(), total_norm_before.item(), total_norm_after.item()


# ─────────────────────────────────────────────────────────────────────────
# 5. PyTorch built-in RNN vs our manual implementation
# ─────────────────────────────────────────────────────────────────────────
batch, seq_len, input_sz, hidden_sz = 4, 10, 8, 16
x = torch.randn(batch, seq_len, input_sz)

# Our implementation
manual_rnn = SimpleRNN(input_sz, hidden_sz, output_size=2)
out_manual, hidden_manual = manual_rnn(x)

# PyTorch built-in (batch_first=True: (batch, seq, feature) convention)
pytorch_rnn = nn.RNN(input_sz, hidden_sz, batch_first=True, nonlinearity='tanh')
out_pt, h_n_pt = pytorch_rnn(x)

print(f"Manual  RNN — all hidden states: {hidden_manual.shape}")
print(f"PyTorch RNN — all hidden states: {out_pt.shape}")
print(f"PyTorch RNN — final hidden state: {h_n_pt.shape}")
print()

# Gradient clipping demo
model  = SimpleRNN(input_sz, hidden_sz, 2)
optim  = torch.optim.Adam(model.parameters(), lr=1e-3)
x_long = torch.randn(4, 50, input_sz)
y      = torch.randint(0, 2, (4,))
loss, norm_before, norm_after = training_step_with_clipping(model, x_long, y, optim, clip_norm=1.0)
print(f"Loss: {loss:.4f} | Grad norm before clip: {norm_before:.4f} | after clip: {norm_after:.4f}")
print(f"Clipping {'applied' if norm_before > 1.0 else 'not needed'} (threshold=1.0)")

# Vanishing gradient demo
print()
demonstrate_vanishing_gradient([5, 20, 50, 100])

08 — Where This Leads: The Path to LSTM and Attention

The RNN gave us something genuinely new: a model that reads sequences and carries memory. But its fundamental limitation is now mathematically precise — not a vague complaint, but a provable consequence of chaining multiplications through time. The gradient literally cannot reach the beginning of long sequences.

The history of sequence modelling from 1997 to 2017 is essentially the story of solving this one problem:

09 — A Thought to Sit With

10 — Coming Up Next

Tomorrow — Day 21: GRU & LSTM. We take the RNN's core weakness — the vanishing gradient through recurrent multiplications — and solve it architecturally. The LSTM introduces a cell state and three gates: forget, input, and output. The GRU simplifies this to two gates. We'll build both from scratch, trace gradient flow through the cell state highway, and understand exactly why these architectures lasted 20 years and still appear in production systems today.

11 — Go Deeper

📄 Learning Representations by Back-propagating Errors — Rumelhart, Hinton & Williams (1986)
The paper that introduced backpropagation formally. The foundation of everything in this post.

📄 On the Difficulty of Training Recurrent Neural Networks — Pascanu et al. (2013)
The canonical treatment of vanishing and exploding gradients in RNNs. Section 2 contains the mathematical proof. Section 4 describes gradient clipping — still used today.

▶ Recurrent Neural Networks (RNN) and Long Short-Term Memory (LSTM) — StatQuest
YouTube · 19 min · The clearest visual explanation of how hidden states carry information through time. Watch before tackling the LSTM.

🔗 The Unreasonable Effectiveness of Recurrent Neural Networks — Andrej Karpathy (2015)
The most famous blog post on RNNs. Generates Shakespeare, Linux kernel code, and LaTeX papers character by character. Still a joy to read — and reveals what RNNs are genuinely good at.

📄 Sequence to Sequence Learning with Neural Networks — Sutskever et al. (2014)
The seq2seq paper. Introduced the encoder-decoder architecture. Google Translate ran on this. Section 2 is the whole architecture in two pages.