Transfer Learning & Fine-Tuning:
Standing on the Shoulders of Millions of Images

A newly graduated radiologist doesn't start their first day knowing nothing.
They bring twelve years of education, thousands of hours of anatomy,
an entire medical school's worth of pattern recognition.
On their first day, they apply all of that knowledge to the specific cases in front of them.

A pre-trained neural network is that radiologist on day one.
It already knows how to see. You just need to show it what to look for.

01 — The Core Idea: Don't Start from Zero

Training a powerful neural network from scratch requires enormous amounts of labelled data. Most real-world problems don't have that luxury. Medical imaging datasets are expensive to annotate — expert radiologists must label each scan. Legal document classifiers need expert review. Industrial defect detectors rely on rare failure examples. The most valuable tasks are often exactly where data is scarcest.

Transfer learning is the solution. Instead of initialising weights randomly and training entirely on your small dataset, you start from a model already trained on a large, related dataset — and transfer its learned knowledge to your task. You keep what it already knows, then adapt it to what you need.

This isn't a computer vision trick. It's the operating principle behind the entire modern AI landscape. GPT-4, Claude, Gemini — all pre-trained foundation models fine-tuned for specific tasks. The CNN fine-tuning we'll use as an example today and the LoRA fine-tuning of LLMs in Day 25 are the same idea at different scales.

02 — What Fine-Tuning Actually Means

Fine-tuning is taking a pre-trained model and continuing to train it — partially or fully — on a new, smaller dataset. The key is that not all layers need the same treatment. Layers that learned general knowledge should barely change. Layers that learned task-specific patterns need more adaptation. The output head, trained for the original task's classes, must be replaced entirely.

03 — Feature Extraction vs Fine-Tuning

Transfer learning has two operating modes. The right choice depends on how different your task is from the pre-training task, and how much data you have.

In practice, do both — sequentially. Start frozen (head only) for 5–10 epochs until the head converges. Then unfreeze and fine-tune at a much smaller learning rate. Skipping Phase 1 is dangerous: a randomly-initialised head produces large gradients that flow backward and overwrite the pre-trained backbone in the very first updates.

04 — Data Augmentation: The Other Lever

Transfer learning solves the "not enough data to train deep features" problem. Augmentation solves the second problem: not enough data to prevent the new head from memorising your small training set.

Augmentation teaches invariances — the label shouldn't change when you rotate, flip, or adjust brightness. You're enforcing: "a glioma is still a glioma at 15 degrees." Each augmented view isn't new ground truth — it's a different perspective on the same example. Over many epochs with a few stochastic transforms, a 3,000-image dataset can produce ~360,000 distinct training views: a 400% effective expansion at zero labelling cost.

One rule is absolute: augmentation is for training only. Validation transforms must be deterministic. Augmenting validation makes accuracy non-deterministic across runs, which makes overfitting undetectable.

05 — The General Workflow

06 — CNN Example: Brain Tumour Classifier at 95%

Here's the workflow applied to a concrete case — the brain tumour project from yesterday's CNN post. The dataset has 3,000 MRI scans (glioma, meningioma, pituitary, no tumour). Too few to train a deep CNN from scratch without severe overfitting. A plain CNN trained from random weights achieves ~63%. With a pre-trained ResNet50 and the two-phase strategy, it reaches 95.5%. The 32-point gap is entirely borrowed knowledge.

import torch
import torch.nn as nn
import torchvision
from torchvision import transforms, datasets
from torch.utils.data import DataLoader
import numpy as np

# ── Hyperparameters ──────────────────────────────────────────────────
NUM_CLASSES = 4          # glioma, meningioma, pituitary, no_tumor
IMG_SIZE    = 224
BATCH_SIZE  = 32
HEAD_LR     = 1e-3       # Phase 1: head-only training
FINETUNE_LR = 1e-5       # Phase 2: full fine-tuning
HEAD_EPOCHS = 8          # Phase 1 epochs
FT_EPOCHS   = 22         # Phase 2 epochs
DEVICE      = 'cuda' if torch.cuda.is_available() else 'cpu'

# ── Data Transforms ──────────────────────────────────────────────────
train_tfm = transforms.Compose([
    transforms.Resize(256),
    transforms.RandomCrop(IMG_SIZE),
    transforms.RandomHorizontalFlip(),
    transforms.RandomVerticalFlip(p=0.3),
    transforms.RandomRotation(20),
    transforms.ColorJitter(brightness=0.4, contrast=0.4),
    transforms.ToTensor(),
    transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
])
val_tfm = transforms.Compose([
    transforms.Resize(256), transforms.CenterCrop(IMG_SIZE),
    transforms.ToTensor(),
    transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
])

# ── Data Loaders ─────────────────────────────────────────────────────
# Dataset structure: data/train/{glioma,meningioma,pituitary,no_tumor}/
train_ds = datasets.ImageFolder('data/train', transform=train_tfm)
val_ds   = datasets.ImageFolder('data/val',   transform=val_tfm)
train_dl = DataLoader(train_ds, batch_size=BATCH_SIZE, shuffle=True,  num_workers=4)
val_dl   = DataLoader(val_ds,   batch_size=BATCH_SIZE, shuffle=False, num_workers=4)
print(f"Train: {len(train_ds)} | Val: {len(val_ds)}")
print(f"Classes: {train_ds.classes}")

# ── Model Definition ─────────────────────────────────────────────────
class TumourClassifier(nn.Module):
    def __init__(self, num_classes=NUM_CLASSES):
        super().__init__()
        # Load ResNet50 with ImageNet weights — NEVER use random init for backbone
        self.backbone = torchvision.models.resnet50(weights='IMAGENET1K_V2')
        in_features = self.backbone.fc.in_features  # 2048 for ResNet50

        # Replace final layer with task-specific head
        self.backbone.fc = nn.Sequential(
            nn.Linear(in_features, 256),
            nn.ReLU(),
            nn.Dropout(0.4),
            nn.Linear(256, num_classes)
        )

    def freeze_backbone(self):
        """Phase 1: freeze everything except the new head."""
        for name, param in self.backbone.named_parameters():
            if 'fc' not in name:  # 'fc' is our new head
                param.requires_grad = False

    def unfreeze_backbone(self):
        """Phase 2: enable gradients for all layers."""
        for param in self.backbone.parameters():
            param.requires_grad = True

    def trainable_params(self):
        return sum(p.numel() for p in self.parameters() if p.requires_grad)

    def forward(self, x):
        return self.backbone(x)

# ── Training Helper ──────────────────────────────────────────────────
def run_epoch(model, loader, criterion, optimizer=None):
    training = optimizer is not None
    model.train() if training else model.eval()
    total_loss, correct = 0.0, 0
    ctx = torch.enable_grad() if training else torch.no_grad()
    with ctx:
        for imgs, labels in loader:
            imgs, labels = imgs.to(DEVICE), labels.to(DEVICE)
            logits = model(imgs)
            loss = criterion(logits, labels)
            if training:
                optimizer.zero_grad(); loss.backward(); optimizer.step()
            total_loss += loss.item() * imgs.size(0)
            correct += (logits.argmax(1) == labels).sum().item()
    n = len(loader.dataset)
    return total_loss / n, correct / n

# ── Main Training Loop ───────────────────────────────────────────────
model = TumourClassifier().to(DEVICE)
criterion = nn.CrossEntropyLoss()

# ─── PHASE 1: Head-only training (frozen backbone) ───────────────────
print("\n=== Phase 1: Head Training (frozen backbone) ===")
model.freeze_backbone()
print(f"Trainable params: {model.trainable_params():,}")
optimizer = torch.optim.AdamW(
    filter(lambda p: p.requires_grad, model.parameters()), lr=HEAD_LR
)
for epoch in range(1, HEAD_EPOCHS + 1):
    tr_loss, tr_acc = run_epoch(model, train_dl, criterion, optimizer)
    va_loss, va_acc = run_epoch(model, val_dl, criterion)
    if epoch % 2 == 0:
        print(f"  ep {epoch:02d} | tr_acc={tr_acc:.4f} | va_acc={va_acc:.4f} | va_loss={va_loss:.4f}")

# ─── PHASE 2: Fine-tuning (unfrozen backbone, tiny lr) ───────────────
print("\n=== Phase 2: Fine-Tuning (unfrozen backbone) ===")
model.unfreeze_backbone()
print(f"Trainable params: {model.trainable_params():,}")
optimizer = torch.optim.AdamW(model.parameters(), lr=FINETUNE_LR, weight_decay=1e-4)
scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=FT_EPOCHS)
best_acc, best_state = 0.0, None
for epoch in range(1, FT_EPOCHS + 1):
    tr_loss, tr_acc = run_epoch(model, train_dl, criterion, optimizer)
    va_loss, va_acc = run_epoch(model, val_dl, criterion)
    scheduler.step()
    if va_acc > best_acc:
        best_acc = va_acc
        best_state = {k: v.cpu().clone() for k, v in model.state_dict().items()}
    if epoch % 5 == 0:
        print(f"  ep {epoch:02d} | tr_acc={tr_acc:.4f} | va_acc={va_acc:.4f} | best={best_acc:.4f}")

print(f"\nBest val accuracy: {best_acc:.4f}")

07 — The Decision Framework: Which Approach for Which Situation

Transfer learning isn't one-size-fits-all. The right approach depends on two axes: how similar your target domain is to the pre-training domain, and how much labelled data you have. Every combination has a recommended strategy.

08 — The Same Idea Powers the Entire LLM Ecosystem

Transfer learning for CNNs is not a historical curiosity — it is the foundational concept underlying modern language AI, and understanding the vision version gives you deep intuition for why language fine-tuning works.

09 — A Thought to Sit With

10 — Coming Up Next

Tomorrow — Day 18: Object Detection, Segmentation & YOLO. Classification answers "what is in this image?" Detection answers "where is it, and what is it?" Segmentation goes further: "which exact pixels belong to it?" These three tasks form the hierarchy of computer vision understanding — and YOLO made real-time detection possible by reframing detection as a single-shot regression problem rather than a two-stage pipeline.

11 — Go Deeper

▶ Transfer Learning with PyTorch — Aladdin Persson
YouTube · 30 min · Full practical walkthrough of feature extraction and fine-tuning on a real dataset.

🔗 Transfer Learning for Computer Vision — PyTorch Official Tutorial
Complete code walkthrough training ResNet on a real 2-class dataset with visualisation.

📄 How Transferable are Features in Deep Neural Networks? — Yosinski et al. (2014)
The foundational paper quantifying exactly which layers transfer well and which don't — the experimental basis for everything in §02.

🔗 Hugging Face Model Hub — timm (PyTorch Image Models)
Ross Wightman's library with 700+ pre-trained vision models: pip install timm then timm.create_model('efficientnet_b3', pretrained=True).