A student studying for exams memorizes every question from past papers
word-for-word. On exam day, every question is slightly different —
and they fail completely. They over-optimized for the training data.
Their teacher adds a rule: "Don't just memorize. Understand the concept.
Keep your notes brief." That rule — penalizing unnecessary complexity —
is regularization. And it's built into every model you will ever train.
01 — What Regularization Does
Every ML model learns by minimising a loss function. By default, the only thing the model cares about is getting training examples right. Given enough capacity, it will memorise noise, outliers, and quirks — and fail on anything new.
Regularization adds a second objective: stay simple. The modified loss the model now minimises is:
Total Loss = Prediction Error + λ × Complexity Penalty
The model is now optimising two competing goals simultaneously. It wants small prediction error (fit the data) and small complexity penalty (stay simple). λ (lambda) is the dial between them:
— λ = 0: no regularization , model fits freely, high variance risk.
— λ very large: penalty dominates, model ignores data, high bias risk.
— λ tuned right: the sweet spot — trades a small rise in training error for a large drop in test error.
Worked intuition: suppose a model has weights [3.2, 0.1, 4.7] for three features. The L2 penalty adds 3.2² + 0.1² + 4.7² = 10.24 + 0.01 + 22.09 = 32.34 to the loss. Gradient descent now pushes weights downward not just to reduce prediction error but also to reduce this penalty — shrinking them toward zero. A model that assigns weight 4.7 to a feature is paying 22 units of penalty just for that one weight. It will only keep that weight large if the feature is genuinely earning its cost in prediction accuracy.
You're hiring someone and their CV claims perfect mastery of 15 languages, 30 frameworks, concert-level piano, and Michelin-star cooking. You're suspicious — this level of claimed expertise across everything usually means genuine depth in nothing. You penalise CV complexity and prefer a candidate who says "I'm genuinely excellent at three specific things."
Regularization does exactly this to weights. A model claiming high sensitivity to 100 features pays a large penalty. It will only keep large weights on features that genuinely earn them by reducing prediction error. The result: fewer, stronger, more trustworthy signals.
02 — L2 Regularization (Ridge): Shrink Everything Smoothly
L2 adds the sum of squared weights to the loss:
Loss = MSE + λ × Σ(wᵢ²)
Squaring the weights has two consequences. First, large weights are penalised disproportionately — a weight of 4.7 contributes 22.09 to the penalty, while a weight of 0.1 contributes only 0.01. This aggressively discourages outlier weights. Second, the penalty surface is smooth everywhere — gradient descent slides weights steadily downward but never has a reason to land exactly at zero. All weights survive; large ones shrink dramatically, small ones barely move.
— When features are correlated — Ridge distributes weight across correlated features stably, while L1 picks one arbitrarily.
— As the default first choice for regression with many features.
— sklearn:
Ridge(alpha=1.0). Start at α=1.0, tune up/down with RidgeCV.
03 — L1 Regularization (Lasso): Automatic Feature Selection
L1 adds the sum of absolute weights to the loss:
Loss = MSE + λ × Σ|wᵢ|
Absolute value instead of squaring changes everything geometrically. The penalty shrinks all weights by the same constant amount per gradient step (not proportional to their size). Small weights hit zero first and stay there — gradient descent has no reason to push them back up once they reach zero, because the absolute value function has a kink at zero with equal slope on both sides. This is why L1 produces exact zeros where L2 produces small-but-nonzero weights.
— When you need an interpretable, sparse model — reading 10 surviving weights is easier than reading 100 tiny ones.
— Text classification: 100,000 word features → L1 zeros 99,000 irrelevant ones, keeps only the predictive words.
— sklearn:
Lasso(alpha=0.1). Lower α than Ridge equivalent (different scale).
04 — Elastic Net: The Best of Both
Elastic Net combines L1 and L2 into one penalty. The l1_ratio parameter controls the mix: 0 = pure Ridge, 1 = pure Lasso, 0.5 = equal blend.
In practice: Elastic Net gets the sparsity of L1 (zeros out irrelevant features) combined with the stability of L2 (handles correlated features gracefully). Use it when you suspect some features are irrelevant but also have groups of correlated features — L1 alone would arbitrarily pick one from each correlated group, Elastic Net keeps them all at reduced weight instead.
Ridge(alpha=λ)Lasso(alpha=λ)ElasticNet(alpha=λ, l1_ratio=ρ)Use
RidgeCV / LassoCV to search automatically. Always scale features first (StandardScaler) — the penalty is sensitive to feature magnitude.
05 — From Scratch: Seeing the Penalty Work
Before sklearn abstracts it away, see the actual penalty changing the gradient update. This is what Ridge regularization looks like at the math level — one extra term added to the weight gradient:
import numpy as np
# Tiny dataset: y = 2*x + noise
np.random.seed(42)
X = np.random.randn(20, 1)
y = 2 * X.squeeze() + np.random.randn(20) * 0.5
def ridge_from_scratch(X, y, lam, lr=0.05, epochs=200):
w = 5.0 # start with a large weight to see shrinkage
b = 0.0
for _ in range(epochs):
pred = w * X.squeeze() + b
err = pred - y
# gradient of MSE + L2 penalty with respect to w:
# dMSE/dw = (2/n) * X.T @ err
# dL2/dw = 2 * lam * w ← the extra pull toward zero
dw = (2 / len(y)) * (X.squeeze() @ err) + 2 * lam * w
db = (2 / len(y)) * err.sum()
w -= lr * dw
b -= lr * db
return w, b
print("λ=0.0 (no regularization ): w =", f"{ridge_from_scratch(X, y, lam=0.0)[0]:.4f}")
print("λ=0.5 (light): w =", f"{ridge_from_scratch(X, y, lam=0.5)[0]:.4f}")
print("λ=5.0 (heavy): w =", f"{ridge_from_scratch(X, y, lam=5.0)[0]:.4f}")
print("True weight: w = 2.0000")The single extra term 2 * lam * w in the gradient is the entire L2 regularization . It creates an additional pull toward zero proportional to the current weight — large weights feel a stronger pull than small weights. At λ=0, the model recovers the true weight almost exactly (1.98 ≈ 2.0). At λ=0.5, light shrinkage: still close (1.83). At λ=5.0, the penalty dominates — the model is pulled far from the truth (0.69). This is over-regularization : you have introduced so much bias that the model can no longer learn the true signal. The goal is the λ that minimises test error — found via cross-validation, not guessing.
06 — sklearn: All Three Methods Side by Side
The same problem — 100 features, only 10 truly predictive — run through all four options. Watch which ones zero out the irrelevant 90 features and which one collapses without regularization :
import numpy as np
from sklearn.linear_model import LinearRegression, Ridge, Lasso, ElasticNet, RidgeCV
from sklearn.datasets import make_regression
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
# 100 features, only 10 informative — classic noisy setup
X, y = make_regression(n_samples=200, n_features=100, n_informative=10,
noise=30, random_state=42)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, random_state=42)
# Always scale before regularization — penalty is magnitude-sensitive
sc = StandardScaler()
X_tr = sc.fit_transform(X_tr)
X_te = sc.transform(X_te)
models = [
('No regularization ', LinearRegression()),
('Ridge (L2, α=1.0) ', Ridge(alpha=1.0)),
('Lasso (L1, α=0.1) ', Lasso(alpha=0.1)),
('ElasticNet (α=0.1) ', ElasticNet(alpha=0.1, l1_ratio=0.5)),
]
print(f"{'Model':<22} {'Train R²':>9} {'Test R²':>8} {'Zero weights':>13}")
print("-" * 56)
for name, model in models:
model.fit(X_tr, y_tr)
zeros = np.sum(np.abs(model.coef_) < 1e-4)
print(f"{name} {model.score(X_tr,y_tr):>9.3f} {model.score(X_te,y_te):>8.3f} {zeros:>10}/100")
# Auto-tune Ridge using cross-validation
cv_ridge = RidgeCV(alphas=[0.01, 0.1, 1, 10, 100]).fit(X_tr, y_tr)
print(f"\nRidgeCV best alpha: {cv_ridge.alpha_} Test R²: {cv_ridge.score(X_te,y_te):.3f}")1 — No regularization is catastrophic. Train R²=0.998 (near-perfect memorisation), Test R²=0.541 (barely better than constant prediction). Classic high-variance overfitting — 200 examples, 100 features. The model fitted noise in 90 irrelevant features.
2 — Ridge recovers strongly without zeroing anything. Test R² jumps from 0.541 to 0.913 while keeping all 100 weights nonzero. It shrank the 90 irrelevant weights toward zero but not to zero — they contribute a tiny amount but no longer dominate.
3 — Lasso zeros 83 of the 90 irrelevant features. Test R²=0.902, essentially matching Ridge, but with only 17 nonzero weights. Far more interpretable. In this setup, Lasso effectively recovered the true signal structure automatically.
4 — RidgeCV found a better α automatically. At α=10 (ten times the manual guess), Test R² improves to 0.921. Always use cross-validated search — the right α varies enormously by problem and data scale.
07 — Dropout: Regularization for Neural Networks
L1 and L2 regularise by penalising weight magnitude. Neural networks have an additional problem: individual neurons can co-adapt — one neuron learns to fix another's errors, creating brittle dependencies. Remove that one neuron and everything breaks. Dropout breaks these dependencies by randomly disabling neurons during training.
At each training step, every neuron is independently switched off with probability p (the dropout rate). The neuron outputs zero — it contributes nothing to that forward pass. The model cannot rely on any neuron always being present, so it learns multiple, independent representations of the same pattern. At inference time, all neurons are active but their outputs are scaled by (1−p) to compensate for the extra neurons now contributing.
import torch
import torch.nn as nn
# Network WITHOUT dropout
model_no_drop = nn.Sequential(
nn.Linear(30, 128), nn.ReLU(),
nn.Linear(128, 64), nn.ReLU(),
nn.Linear(64, 1), nn.Sigmoid()
)
# Network WITH dropout — regularised
model_drop = nn.Sequential(
nn.Linear(30, 128),
nn.ReLU(),
nn.Dropout(p=0.3), # 30% of neurons randomly off each step
nn.Linear(128, 64),
nn.ReLU(),
nn.Dropout(p=0.2), # 20% in the second hidden layer
nn.Linear(64, 1),
nn.Sigmoid()
)
# CRITICAL: always switch mode before training / inference
model_drop.train() # activates dropout — neurons randomly disabled
model_drop.eval() # deactivates dropout — all neurons active (forgetting this is a common bug)
print("Parameters without dropout:", sum(p.numel() for p in model_no_drop.parameters()))
print("Parameters with dropout:", sum(p.numel() for p in model_drop.parameters()))
print("Dropout adds no parameters — only stochastic behaviour during training.")p=0.5 — the original Hinton recommendation; use on large networks with clear overfitting.
p=0.0 — effectively disabled; start at 0, increase if train/val gap appears.
Never apply dropout to the output layer. Rarely to the input layer (p ≤ 0.1 if at all).
Most common bug: forgetting
model.eval() before inference — leaves dropout active, producing different predictions each run.
08 — Early Stopping: The Simplest Regularization
No penalty, no probability — just watch the validation loss and stop when it starts climbing. The model has not yet had time to memorise noise. Early stopping is the most widely used regularization technique in deep learning, always used in combination with L2 and dropout.
# Simulated training: val_loss decreases then rises (overfitting begins)
train_losses = [0.90, 0.65, 0.48, 0.35, 0.25, 0.18, 0.13, 0.10, 0.08, 0.06]
val_losses = [0.91, 0.67, 0.51, 0.39, 0.32, 0.31, 0.33, 0.37, 0.42, 0.48]
def early_stopping(val_losses, patience=2):
best_loss, best_epoch, wait = float('inf'), 0, 0
for epoch, loss in enumerate(val_losses, 1):
if loss < best_loss:
best_loss, best_epoch, wait = loss, epoch, 0
else:
wait += 1
if wait >= patience:
print(f"Stop at epoch {epoch}. Best: epoch {best_epoch}, val_loss={best_loss:.2f}")
return
early_stopping(val_losses, patience=2)
for i, (tr, vl) in enumerate(zip(train_losses, val_losses), 1):
tag = " ← best val" if vl == min(val_losses) else (" ← STOP" if i == 8 else "")
print(f" Epoch {i:2d}: train={tr:.2f} val={vl:.2f}{tag}")The best model is at epoch 6 — val_loss=0.31. After that, training loss keeps falling (the model is still improving on training data) but validation loss rises (it is now memorising noise, not learning patterns). With patience=2, the algorithm fires at epoch 8 — two steps after the best. In practice: save model weights at every new best, then restore those saved weights when stopping. Do not use the weights at the stopping epoch — use the best-epoch weights.
09 — Weight Decay and AdamW: Regularization in Modern LLMs
For standard gradient descent, L2 regularization and "weight decay" are two names for the same thing — both shrink weights a little every step. Most textbooks use them interchangeably.
But the popular Adam optimiser broke that equivalence. Adam is smarter than plain gradient descent — it tracks the history of each weight's gradients and adjusts the update size per-weight accordingly. Weights that have been jumping around get smaller updates; weights that have been moving steadily get larger ones. The problem: when you add an L2 penalty inside this process, Adam's per-weight adjustments also distort the regularization . Some weights end up over-penalised, others barely penalised at all — you can't predict which without inspecting the gradient history.
AdamW fixes this with a simple change: it applies the weight shrinkage as a completely separate step, after Adam has finished its update. The two operations no longer interfere with each other. Every weight shrinks by the same fraction every step — predictable, consistent, independent of gradient history.
PyTorch: torch.optim.AdamW(model.parameters(), lr=1e-4, weight_decay=0.01)
AdamW is now the standard optimiser for every major LLM — GPT, LLaMA, Mistral, Claude. weight_decay=0.01 is the standard starting point. Dropout was used in BERT and early GPT. LayerNorm in modern Transformers adds a further implicit stabilising effect. Every technique from this post is running inside the models you use daily.
10 — A Thought to Sit With
Regularization penalises complexity — it prefers simpler models. But "simple" is a design choice baked into the penalty function. L1 defines simplicity as "few non-zero weights." L2 defines it as "small weights." Neither definition asks whether the features that survive are causally meaningful or merely correlated with the target.
What kinds of true patterns might L1 accidentally remove? What kinds of spurious correlations might L2 preserve at small magnitude but never zero out? Is a sparse model always more interpretable — or just easier to read?
11 — Coming Up Next
Day 11 — Bagging & Random Forests. Now that you understand variance and the tools that control it, Random Forests will make immediate sense: they reduce variance by averaging many high-variance decision trees. The wisdom-of-crowds principle, made rigorous — and the most reliable ML algorithm for tabular data.
12 — Go Deeper
▶ StatQuest: Ridge, Lasso, and Elastic Net Regression
YouTube · 20 min · The clearest visual explanation of the L1 diamond vs L2 circle geometry — the intuition you need to truly understand why they behave differently.
▶ StatQuest: Regularization Part 2 — Lasso Regression
YouTube · 8 min · Why L1 produces exact zeros — the geometric explanation of the diamond's corners landing on the axes.
▶ Decoupled Weight Decay Regularization — Loshchilov & Hutter (2019)
The paper that introduced AdamW — explains precisely why L2 regularization and weight decay diverge under Adam, and demonstrates the empirical improvement. Surprisingly readable for a research paper.
- The core idea: Loss = Prediction Error + λ × Complexity Penalty. Two competing objectives force the model to stay simple while fitting the data.
- λ (alpha) is the key dial. Too low → still overfitting. Too high → underfitting. Always tune with RidgeCV / LassoCV. Always scale features first.
- L2 (Ridge): penalty = Σ(wᵢ²). Shrinks all weights smoothly. Never zeros. Best for correlated features or when all features may be relevant. Default choice.
- L1 (Lasso): penalty = Σ|wᵢ|. Zeros irrelevant weights exactly. Automatic feature selection. Best when many features are noise. Sparse, interpretable result.
- Elastic Net: combines L1 + L2. l1_ratio controls the mix. Best for correlated feature groups with mixed relevance.
- The geometry: L2 circle → smooth boundary, weights never hit exactly zero. L1 diamond → corners on axes, solution lands at zero. This shape explains all the behaviour.
- Dropout: randomly disables neurons (probability p) during training. Breaks co-adaptation. Ensemble interpretation: 2ⁿ sub-networks trained simultaneously. Always model.eval() at inference.
- Early stopping: stop when validation loss rises. Save best-epoch weights. Simplest regularization — use it for every neural network as a baseline.
- Weight decay ≠ L2 under Adam. AdamW decouples weight decay from adaptive gradient scaling, giving consistent regularization . Standard for all LLM training.
- In LLMs: AdamW with weight_decay=0.01–0.1 is universal. GPT, LLaMA, Mistral, Claude all use it. These concepts at scale.
- What is regularization and why do we need it?
Regularization adds a complexity penalty to the loss function: Loss = Prediction Error + λ × Penalty. It forces the model to minimise both error and complexity simultaneously, preventing memorisation of training noise. It trades a small rise in training error for a large drop in test error — deliberately introducing bias to control variance. - What is the difference between L1 and L2 regularization ?
L2 (Ridge) penalises Σ(wᵢ²) — squared weights. It pulls weights toward zero in proportion to their size: large weights get pulled hard, tiny weights barely move. Because the pull weakens as weights shrink, they approach zero but never snap to it. L1 (Lasso) penalises Σ|wᵢ| — absolute weights. It subtracts a fixed constant from every weight each step. Small weights run out of value and hit zero first — and once they're at zero they stay there. This is why L1 produces exact zeros (feature selection) and L2 does not. L2 handles correlated features better; L1 picks one from a correlated group arbitrarily. - When would you use Lasso over Ridge?
When many features are likely irrelevant and you want an interpretable sparse model. Lasso zeros irrelevant weights automatically. Ridge is better when features are correlated (L2 distributes weight stably; L1 picks arbitrarily). For the general case, try both with cross-validated alpha and let test performance decide. - What is dropout and why does it work?
Dropout randomly zeroes each neuron's output with probability p during training. It prevents co-adaptation — neurons cannot rely on specific partners always being present. The ensemble interpretation: 2ⁿ sub-networks trained simultaneously with shared weights, approximating an ensemble average at inference. Always call model.eval() before inference — this deactivates dropout and applies the (1−p) output scaling compensation. - What is AdamW and how does it relate to L2 regularization ?
For SGD, L2 regularization and weight decay are mathematically identical. For Adam, they're not — adding L2 inside Adam's gradient means Adam's adaptive scaling also scales the regularization , making it inconsistent across parameters. AdamW applies weight decay as a separate step after the adaptive gradient update, giving consistent regularization for every parameter. It is the standard optimiser for all major LLM training. Typical weight_decay: 0.01–0.1.
No attachments for this topic yet.