CNNs from Scratch:
How Machines Learned to See

In September 2012, a team from the University of Toronto
submitted an entry to the ImageNet Large Scale Visual Recognition Challenge.
The competition had run since 2010. The best error rate to that point: around 25%.

AlexNet scored 15.3%. The second-place team: 26.2%.

A gap that size doesn't happen in a mature competition.
The judges assumed a mistake had been made. It hadn't.
A paradigm had shifted — and everything that followed,
from self-driving cars to medical AI, traces back to that moment.

01 — The Problem with Feeding Images to a Dense Network

Before building a CNN, it's worth understanding exactly why the obvious approach — feeding raw pixel values into the dense neural network from Day 15 — fails so badly on images. The failure has two distinct causes, and the CNN solves both simultaneously.

Problem 1: The parameter explosion. A colour image at 224×224 resolution contains 224 × 224 × 3 = 150,528 numbers. A dense hidden layer with 1,000 neurons needs one weight per input-output pair: 150,528 × 1,000 = 150.5 million parameters — for a single layer. That's not just slow to train. It's statistically hopeless. With millions of parameters, you need tens of millions of training images just to avoid catastrophic overfitting, before even adding more layers. The entire ImageNet training set has about 1.2 million images. A single dense layer already needs more parameters than can be meaningfully trained on it.

Problem 2: Spatial blindness. In a dense layer, every input neuron connects to every output neuron with a unique, independent weight. There is no concept of "pixel (12, 15) is adjacent to pixel (12, 16)." The network has to learn spatial relationships from scratch, from weights — and it has to re-learn them independently for every possible position. An eye in the top-left corner and an eye in the bottom-right corner of an image are structurally identical, but a dense network treats them as completely different phenomena requiring completely different weights. This is wildly inefficient for any data where spatial structure matters.

import torch
import torch.nn as nn

# ── Demonstrate the parameter gap ────────────────────────────────────────
# Image: 224×224 colour (standard ImageNet input)
img_pixels = 224 * 224 * 3      # = 150,528
hidden_size = 1000

# Dense MLP — first hidden layer only
dense_layer = nn.Linear(img_pixels, hidden_size)
dense_params = sum(p.numel() for p in dense_layer.parameters())

# CNN — first conv layer: 3 input channels, 64 filters, 3×3 kernel
# (same as AlexNet's first layer configuration, but 3×3 not 11×11)
conv_layer  = nn.Conv2d(3, 64, kernel_size=3, padding=1)
conv_params = sum(p.numel() for p in conv_layer.parameters())

print(f"Image size:            {img_pixels:>10,} numbers  (224×224×3)")
print(f"Dense layer params:    {dense_params:>10,}  (150,528 × 1,000 + bias)")
print(f"Conv layer params:     {conv_params:>10,}  (3×3×3×64 + 64 bias)")
print(f"Ratio:                 {dense_params/conv_params:>10.0f}×  more params in dense")
print()

# Full network comparison — 3-layer dense MLP vs 3-block CNN
dense_mlp = nn.Sequential(
    nn.Flatten(),
    nn.Linear(150528, 1000), nn.ReLU(),
    nn.Linear(1000,   512),  nn.ReLU(),
    nn.Linear(512,    10),
)
simple_cnn = nn.Sequential(
    nn.Conv2d(3,  32, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),  # 224→112
    nn.Conv2d(32, 64, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),  # 112→56
    nn.Conv2d(64, 128,3, padding=1), nn.ReLU(), nn.AdaptiveAvgPool2d(1),
    nn.Flatten(),
    nn.Linear(128, 10),
)
mlp_total = sum(p.numel() for p in dense_mlp.parameters())
cnn_total = sum(p.numel() for p in simple_cnn.parameters())

print(f"3-layer dense MLP:     {mlp_total:>10,} parameters")
print(f"3-block CNN:           {cnn_total:>10,} parameters")
print(f"CNN is {mlp_total/cnn_total:.0f}× smaller, yet better at images")

02 — The Convolution Operation, Step by Step

A convolution filter is a small grid of learned numbers — typically 3×3. You place it at the top-left corner of the image, multiply each filter value by the corresponding pixel value underneath it, sum all 9 products, and write that single number to an output grid. Then slide the filter one pixel right and repeat. When you reach the end of the row, drop one row and start again from the left. The complete output is called a feature map — one value per position, representing how strongly the filter's pattern appeared at each location.

import numpy as np

# ── Manual 2D convolution — one filter, one pass ──────────────────────────
# Simulated 5×5 grayscale image patch
image = np.array([
    [10, 10,  0,  0,  0],
    [10, 10,  0,  0,  0],
    [10, 10,  0,  0,  0],
    [10, 10,  0,  0,  0],
    [10, 10,  0,  0,  0],
], dtype=float)   # Bright left half, dark right half — vertical edge at column 2

# Sobel vertical edge detector (this is what CNNs often learn as first filters)
filter_vertical = np.array([
    [-1,  0,  1],
    [-1,  0,  1],
    [-1,  0,  1],
])

# Manual convolution (no padding, stride=1) → output is (5-3+1)×(5-3+1) = 3×3
H, W = image.shape
fH, fW = filter_vertical.shape
out_H, out_W = H - fH + 1, W - fW + 1

feature_map = np.zeros((out_H, out_W))
for i in range(out_H):
    for j in range(out_W):
        patch = image[i:i+fH, j:j+fW]          # 3×3 patch of the image
        feature_map[i, j] = (patch * filter_vertical).sum()  # element-wise multiply + sum

print("Image (bright left, dark right):")
print(image.astype(int))
print("\nVertical-edge filter:")
print(filter_vertical)
print("\nFeature map (response to vertical edge):")
print(feature_map.astype(int))
print("\nInterpretation:")
print("  Column 1 response: large negative (left side of bright region)")
print("  Column 2 response: zero        (inside bright region, no edge)")
print("  Column 3 response: large neg.  (right side — transition zone)")
print("\nAfter abs(ReLU): the filter fires strongly at the vertical edge position")

# Show one computation step manually
print("\n── Manual computation at position (0,1) ──")
patch = image[0:3, 1:4]
print(f"Patch:\n{patch.astype(int)}")
print(f"Filter:\n{filter_vertical}")
products = patch * filter_vertical
print(f"Element-wise products:\n{products.astype(int)}")
print(f"Sum = {products.sum():.0f} → feature_map[0,1]")

03 — The CNN Forward Pass: Layer by Layer

A single convolution produces one feature map. A layer has many filters — say 64 — each looking for a different pattern. Apply all 64 to the same input and you get 64 feature maps stacked together: a 3D volume of shape (64, H, W). This volume is the input to the next layer. Each successive layer detects patterns in the previous layer's patterns — building a hierarchy from simple to complex.

04 — Receptive Fields: What Each Neuron Actually Sees

A neuron in the first conv layer with a 3×3 filter sees a 3×3 patch of the input image. Call this its receptive field — the region of the original input that influences its output. A neuron in the second conv layer sees a 3×3 patch of the first layer's output — but each of those first-layer neurons itself saw a 3×3 patch of the original input. So the second-layer neuron's receptive field in the original image is 5×5. With pooling (which downsamples by 2×), receptive fields grow even faster.

import numpy as np

# Receptive field growth with depth — 3×3 conv, no pooling
print("Receptive field growth through conv layers (3×3 kernel, stride=1):")
print(f"{'Layer':>6}  {'Receptive Field':>17}  {'What it 'sees' in the image':>35}")
print("-" * 65)

rf = 1
descriptions = [
    "raw pixels",
    "micro-textures, single edges",
    "edges, gradients, corners",
    "simple shapes, stripes",
    "curves, compound edges",
    "textures, patterns",
    "object parts (eye, wheel)",
    "full objects, faces",
]
for layer in range(8):
    rf = rf + 2 if layer > 0 else 1
    print(f"{layer:>6}  {rf:>8}×{rf:<8}  {descriptions[layer]:>35}")

print("\nWith 2×2 max-pooling after every 2 conv layers:")
print(f"{'Layer':>6}  {'RF (orig. image)':>17}  {'Notes':>35}")
print("-" * 65)
rf_pool = 1; scale = 1
pool_notes = ["input","3×3 patch","pooling → RF doubles","11×11 effective","pooling again","27×27 — half the face"]
rf_vals = [1,3,5,10,14,28]
for i, (rv, note) in enumerate(zip(rf_vals, pool_notes)):
    print(f"{i:>6}  {rv:>8}×{rv:<8}  {note:>35}")

05 — Batch Normalisation: The Training Stabiliser

Batch Normalisation (BatchNorm), introduced by Ioffe and Szegedy at Google in 2015, is one of the most impactful practical contributions in deep learning history. Before it, training networks deeper than about 20 layers required extremely careful weight initialisation and small learning rates. After it, 50-layer or 100-layer networks trained stably as a matter of routine. Understanding what it does and why it helps is essential.

import numpy as np

# Simulate what BatchNorm does to activations
np.random.seed(42)

# Activations from a conv layer — wildly different scales across channels
# 4 examples, 4 channels, each channel at very different scale
activations = np.array([
    [ 100, 0.01, -50,  200],
    [ 150, 0.02, -30,  180],
    [  80, 0.005,-70,  220],
    [ 120, 0.015,-45,  195],
])
# Columns: 4 channels. Rows: 4 examples in the batch.

# BatchNorm: normalise each channel across the batch
mu  = activations.mean(axis=0)       # mean per channel
var = activations.var(axis=0)        # variance per channel
eps = 1e-8
normed = (activations - mu) / np.sqrt(var + eps)

print("Before BatchNorm — activations (4 examples × 4 channels):")
print(activations)
print(f"\nChannel means:    {mu}")
print(f"Channel std devs: {np.sqrt(var).round(4)}")
print("\nAfter BatchNorm — normalised activations:")
print(normed.round(3))
print(f"\nPost-norm means:    {normed.mean(axis=0).round(6)}  ← all ≈ 0")
print(f"Post-norm std devs: {normed.std(axis=0).round(4)}   ← all ≈ 1")
print("\nGradient magnitude stabilised — all channels now comparable scale.")
print("Channel 2 (0.01–0.02 range) and Channel 1 (80–150 range) now equal footing.")

06 — A Complete CNN: Training on Real Data

Here is a production-structured CNN trained on the CIFAR-10 dataset — 60,000 colour images across 10 classes. Everything from the previous sections comes together: convolutions, BatchNorm, ReLU, pooling, dropout, and a classifier head.

import torch
import torch.nn as nn
import torch.optim as optim
from torchvision import datasets, transforms
from torch.utils.data import DataLoader

# ── Data augmentation + normalisation ────────────────────────────────────
transform_train = transforms.Compose([
    transforms.RandomHorizontalFlip(),                 # augment: flip left/right
    transforms.RandomCrop(32, padding=4),              # augment: crop with padding
    transforms.ToTensor(),
    transforms.Normalize((0.4914,0.4822,0.4465),      # CIFAR-10 mean per channel
                         (0.2470,0.2435,0.2616))     # CIFAR-10 std per channel
])
transform_test = transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize((0.4914,0.4822,0.4465),
                         (0.2470,0.2435,0.2616))
])

train_ds = datasets.CIFAR10(root='./data', train=True,  download=True, transform=transform_train)
test_ds  = datasets.CIFAR10(root='./data', train=False, download=True, transform=transform_test)
train_loader = DataLoader(train_ds, batch_size=128, shuffle=True,  num_workers=2)
test_loader  = DataLoader(test_ds,  batch_size=256, shuffle=False, num_workers=2)

# ── Model: Conv → BN → ReLU → Pool (3 blocks) ────────────────────────────
class CNN(nn.Module):
    def __init__(self):
        super().__init__()

        def conv_block(in_ch, out_ch):
            return nn.Sequential(
                nn.Conv2d(in_ch, out_ch, kernel_size=3, padding=1, bias=False),
                nn.BatchNorm2d(out_ch),   # bias=False in Conv because BN has its own β
                nn.ReLU(inplace=True),
                nn.Conv2d(out_ch, out_ch, kernel_size=3, padding=1, bias=False),
                nn.BatchNorm2d(out_ch),
                nn.ReLU(inplace=True),
                nn.MaxPool2d(2),
            )

        self.features = nn.Sequential(
            conv_block(3,   64),   # 32×32 → 16×16 | 3 channels → 64 feature maps
            conv_block(64,  128),  # 16×16 → 8×8   | 64 → 128 feature maps
            conv_block(128, 256),  # 8×8  → 4×4    | 128 → 256 feature maps
        )
        self.classifier = nn.Sequential(
            nn.AdaptiveAvgPool2d(1),   # 4×4×256 → 1×1×256 (Global Average Pooling)
            nn.Flatten(),               # 1×1×256 → 256
            nn.Linear(256, 256),
            nn.ReLU(inplace=True),
            nn.Dropout(0.4),
            nn.Linear(256, 10),     # 10 CIFAR-10 classes
        )

    def forward(self, x):
        return self.classifier(self.features(x))

model     = CNN().cuda() if torch.cuda.is_available() else CNN()
criterion = nn.CrossEntropyLoss()
optimizer = optim.AdamW(model.parameters(), lr=1e-3, weight_decay=1e-4)
scheduler = optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=30)

n_params = sum(p.numel() for p in model.parameters())
print(f"Parameters: {n_params:,}  |  Device: {next(model.parameters()).device}")

# ── Training loop ──────────────────────────────────────────────────────────
def evaluate(loader):
    model.eval()
    correct = total = 0
    with torch.no_grad():
        for X, y in loader:
            X, y = X.to(next(model.parameters()).device), y.to(next(model.parameters()).device)
            correct += (model(X).argmax(1) == y).sum().item()
            total   += y.size(0)
    return correct / total

print(f"\n{'Epoch':>5} {'Train Loss':>11} {'Train Acc':>10} {'Test Acc':>9}")
print("-" * 42)

for epoch in range(1, 31):
    model.train()
    total_loss = 0
    for X, y in train_loader:
        X, y = X.to(next(model.parameters()).device), y.to(next(model.parameters()).device)
        optimizer.zero_grad()
        loss = criterion(model(X), y)
        loss.backward()
        optimizer.step()
        total_loss += loss.item()
    scheduler.step()

    if epoch % 5 == 0:
        tr_acc = evaluate(train_loader)
        te_acc = evaluate(test_loader)
        print(f"{epoch:>5} {total_loss/len(train_loader):>11.4f} {tr_acc:>10.4f} {te_acc:>9.4f}")

07 — Residual Connections: How ResNet Fixed Deep Networks

By 2014, researchers were stacking more layers hoping for better accuracy. Something unexpected happened: networks with 56 layers were worse than networks with 20 layers — not just on the test set, but on the training set too. This wasn't overfitting (a test-time phenomenon). The deeper network was genuinely failing to learn, even on the data it was trained on.

The cause was the vanishing gradient problem compounded across extreme depth. By the time gradients propagated back to the early layers through 56 non-linear transformations, they had shrunk to near-zero — and those layers barely updated. Kaiming He and colleagues at Microsoft Research published the fix in 2015: residual connections.

import torch
import torch.nn as nn

# ── Residual block (standard ResNet building block) ───────────────────────
class ResidualBlock(nn.Module):
    def __init__(self, channels, downsample=False):
        super().__init__()
        stride = 2 if downsample else 1

        self.block = nn.Sequential(
            nn.Conv2d(channels, channels, 3, stride=stride, padding=1, bias=False),
            nn.BatchNorm2d(channels),
            nn.ReLU(inplace=True),
            nn.Conv2d(channels, channels, 3, padding=1, bias=False),
            nn.BatchNorm2d(channels),
            # Note: NO ReLU here — ReLU applied AFTER the skip addition
        )

        # Skip connection: if downsampling, match spatial and channel dimensions
        self.skip = nn.Sequential(
            nn.Conv2d(channels, channels, 1, stride=stride, bias=False),
            nn.BatchNorm2d(channels),
        ) if downsample else nn.Identity()   # Identity = "add x unchanged"

        self.relu = nn.ReLU(inplace=True)

    def forward(self, x):
        return self.relu(self.block(x) + self.skip(x))
        #                  ↑ F(x)           ↑ x (the skip)   ← THIS is the residual connection

# ── Verify shapes and gradient flow ──────────────────────────────────────
block = ResidualBlock(64)
x = torch.randn(2, 64, 32, 32)   # 2 images, 64 channels, 32×32
out = block(x)

print(f"Input  shape:  {tuple(x.shape)}")
print(f"Output shape:  {tuple(out.shape)}  ← same shape (no spatial change)")
print(f"Block params:  {sum(p.numel() for p in block.parameters()):,}")

# Show gradient magnitude before vs after skip connections
out.mean().backward()
grad_norm_with_skip = x.grad.abs().mean().item() if x.requires_grad else "N/A"

x2 = torch.randn(2, 64, 32, 32, requires_grad=True)
out2 = block(x2)
out2.mean().backward()

print(f"\nGradient at input (with skip):    {x2.grad.abs().mean().item():.6f}")
print(f"Gradient still non-zero — early layers receive training signal")
print(f"\nResNet-50 stacks {8*4 + 6} such blocks → still gets clean gradients at block 1")

08 — The Architecture Evolution: AlexNet to ConvNeXt

Every major CNN architecture from 2012 onwards solved a specific problem in the previous generation. Understanding what each one fixed makes the evolution logical rather than a list to memorise.

import torch
import torch.nn as nn

# Architecture comparison — parameters and approximate FLOPS
architectures = [
    ("AlexNet",       60,    1500,  "15.3% error (2012)"),
    ("VGG-16",        138,   15500, "7.3% error (2014)"),
    ("ResNet-50",     25,    4100,  "6.7% error (2015)"),
    ("MobileNetV2",   3.4,   300,   "6.9% error (2018)"),
    ("EfficientNet-B4",19,  4200,  "3.5% error (2019)"),
    ("ConvNeXt-Base", 89,    15400, "2.5% error (2022)"),
]
print(f"{'Model':>20} {'Params (M)':>12} {'MAdd (M)':>10} {'ImageNet Result':>22}")
print("-" * 72)
for name, params, madds, result in architectures:
    print(f"{name:>20} {params:>12.1f} {madds:>10,} {result:>22}")

09 — CNN vs Dense MLP: The Full Comparison

Now you have everything needed to make the comparison precise and complete. This is the core question of the post — and the answer runs deeper than just "fewer parameters."

10 — What CNNs Learn: Visualising Filters

One of the remarkable facts about CNNs is that their learned filters are interpretable — at least in the early layers. Because the first-layer filters operate directly on pixel values, you can visualise them as images. What you find is striking: without any human guidance, CNNs trained on natural images learn filters that look like Gabor filters — oriented edge detectors at various angles and frequencies — which are also what the primary visual cortex (V1) in mammalian brains responds to.

11 — A Thought to Sit With

12 — Coming Up Next

Day 17 — Transfer Learning & Fine-Tuning. Training a CNN from scratch on a small dataset rarely works — there aren't enough examples to learn meaningful filters. Transfer learning solves this: start from a network pre-trained on ImageNet (millions of images) and adapt its learned filters to your specific problem with just hundreds or thousands of examples. We'll walk through exactly how this is done — and how this technique powered a brain tumour classification project to 95% accuracy with a dataset too small to train from scratch.

13 — Go Deeper

▶ But what is a convolution? — 3Blue1Brown
YouTube · 23 min · The most visual, intuitive explanation of what convolution computes and why. Watch before reading §02.

▶ CNN Explainer — Georgia Tech / Interactive Tool
Interactive browser demo · Watch a live CNN classify your own images while showing the feature maps at every layer in real time. The best tool for building visual intuition about what each layer is doing.

🔗 CS231n: Convolutional Neural Networks — Stanford
Course notes · The definitive deep-dive into CNN architecture design, receptive fields, and the full parameter count analysis. Read after this post for the formal treatment.

📄 Deep Residual Learning for Image Recognition — He et al. (2015)
arXiv · The ResNet paper. Unusually readable for a landmark paper. Sections 1–3 explain the degradation problem and skip connection solution clearly.