Two archers are competing. The first always hits the same spot —
but it's consistently six inches above the bullseye. Precise, but wrong.
The second hits the bullseye on average — but their arrows are scattered
all over the target. Accurate on average, but wildly inconsistent.
The first archer has high bias. The second has high variance.
This is the central tension of every machine learning model ever built.
Understanding it is understanding why models fail — and exactly how to fix them.
01 — Two Ways a Model Can Fail
Every ML failure traces back to one of two problems. There is no third kind. Once you can see which problem you have, you know exactly which fixes to try — and which ones are a waste of time.
The key symptom: the model makes systematic, consistent errors even on its own training data. It's not close to the truth — and giving it more data won't help, because the model's form is the problem, not the amount of data.
Concrete example: You're predicting house prices. The true relationship is nonlinear — small houses and large houses are both cheap, mid-size are expensive (a curve). Linear regression can only draw a straight line. No matter how many houses you train on, it will always be wrong — consistently underestimating mid-size houses, consistently overestimating extreme sizes. The wrong assumption (linearity) is baked in.
The key symptom: the model performs brilliantly on training data (it memorized it) but fails badly on new data. If you retrained it on a slightly different dataset, you'd get a noticeably different model.
Concrete example: A depth-20 decision tree on the same house price problem. It creates hundreds of tiny leaves that perfectly memorize every house in the training set — including the one unusual house that sold cheap because of a divorce. That "pattern" does not generalize. Every new house it encounters falls into a memorized region built for a specific quirk, not a real rule.
02 — The Full Error Equation
Bias and variance aren't just metaphors — they're components of a precise mathematical formula for how much error a model makes on new data:
Total Error = Bias² + Variance + Irreducible Noise
Each term has a precise meaning:
Bias² — How far off is the model's average prediction from the true answer? If you trained the model 100 times on different random samples and averaged all its predictions, how wrong is that average? Linear regression on a nonlinear problem has high Bias² — it's always wrong in the same direction.
Variance — How much do the model's predictions fluctuate across different training sets? If you retrained on a slightly different dataset, how different would the predictions be? A K=1 KNN or a deep decision tree has high Variance — it memorizes whatever it sees.
Irreducible Noise — Error that no model can ever eliminate. The data itself contains randomness — measurement errors, missing features that affect the outcome, inherent unpredictability. Even if you had a perfect model with zero bias and zero variance, this floor remains. You cannot beat irreducible noise by improving your algorithm. This is the fundamental limit of any prediction problem.
The goal: minimize Bias² + Variance. You cannot touch the Noise floor. The tradeoff is that making the model more complex reduces Bias² but increases Variance, and vice versa.
03 — The Dartboard: All Four Cases
Imagine training your model 10 times on 10 different random samples from the same population. Each trained model makes a prediction on the same new point. Where do those 10 predictions land?
What it looks like: Train accuracy ~97%, test accuracy ~96%, small gap. Learning curves converge at a low error level.
ML examples: Random Forest with tuned hyperparameters, XGBoost on tabular data, well-regularized neural network with sufficient data. These don't come free — they require careful tuning.
What it looks like: Train accuracy ~75%, test accuracy ~74%. Both errors are high and similar — almost no gap. Adding more data doesn't help.
ML examples: Linear regression on a nonlinear problem, Logistic regression when classes aren't linearly separable, Naive Bayes when independence assumption is severely violated, Decision tree with max_depth=1. The model's form is the problem.
What it looks like: Train accuracy ~100%, test accuracy ~87%. Large gap. The model memorized training data perfectly but didn't learn the pattern.
ML examples: Unlimited decision tree (depth 20+), KNN with K=1, neural network trained without regularization on small data. The fix is usually regularization, pruning, or more data.
What it looks like: Train accuracy ~80%, test accuracy ~65%. Large gap and both are bad.
ML examples: Rare in practice, but can happen with a very wrong model architecture applied to mismatched data, or a model with both the wrong inductive bias and too much flexibility for its data size. The fix requires rethinking the model choice entirely.
04 — Reading Error Curves: Diagnosing Your Model in Practice
You don't need to run 100 experiments to diagnose bias vs variance. Two numbers — training error and validation/test error — tell you almost everything. Here is the complete diagnostic guide:
→ High Bias (underfitting). The model can't even learn the training data well. The gap is small because the model is equally bad at everything — it's not memorizing, it just fundamentally can't represent the pattern.
Correct fixes: increase model complexity (deeper tree, more neurons, more features), reduce regularization, add polynomial or interaction features, try a more powerful model class entirely.
Incorrect fix: adding more training data — this almost never helps high bias. If the model is too simple, more data won't make it less simple.
→ High Variance (overfitting). The model is doing well on what it memorized but failing on anything new. The gap IS the variance.
Correct fixes: add regularization (L1, L2, Dropout), reduce model complexity (smaller max_depth, fewer neurons/layers), collect more training data, use data augmentation, use ensemble methods (averaging many models cancels out individual overfitting).
Incorrect fix: making the model more complex — this will widen the gap further.
→ Good generalization. You're done. Resist the urge to over-optimize. Every additional tweak risks pushing back into high variance territory. Ship it.
→ Persistent overfitting. Adding data helps variance but has diminishing returns once the model has "seen enough variety." The gap closes slowly. Use regularization in combination with more data — not instead of it.
05 — Demonstrated in Code: High Bias vs High Variance
The same dataset — breast cancer classification — but three different levels of model complexity. Watch what happens to training and test accuracy, and to the gap between them.
import numpy as np
from sklearn.model_selection import learning_curve, train_test_split
from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import load_breast_cancer
X, y = load_breast_cancer(return_X_y=True)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, random_state=42)
configs = [
(1, "depth=1 — extreme underfitting (stump)"),
(4, "depth=4 — well-tuned (sweet spot)"),
(None, "unlimited — extreme overfitting"),
]
print(f"{'Model':<40} {'Train':>7} {'Test':>7} {'Gap':>6} {'Verdict'}")
print("-" * 75)
for depth, label in configs:
model = DecisionTreeClassifier(max_depth=depth, random_state=42).fit(X_tr, y_tr)
train_acc = model.score(X_tr, y_tr)
test_acc = model.score(X_te, y_te)
gap = train_acc - test_acc
if gap < 0.03 and train_acc > 0.93: verdict = "✅ Good"
elif gap < 0.03: verdict = "❌ High Bias"
else: verdict = "❌ High Variance"
print(f"{label:<40} {train_acc:>6.1%} {test_acc:>6.1%} {gap:>5.1%} {verdict}")
# Learning curve: how does test accuracy change with more training data?
print("\nLearning curve — does more data help?")
for depth, label in [(1, "High Bias (depth=1) "), (None, "High Variance (unlimited)")]:
sizes, train_s, val_s = learning_curve(
DecisionTreeClassifier(max_depth=depth, random_state=42), X, y,
train_sizes=[0.2, 0.5, 1.0], cv=5, scoring='accuracy')
print(f" {label}: val @ 20% data={val_s[0].mean():.1%} val @ 100% data={val_s[-1].mean():.1%}")1 — High bias doesn't improve with more data. The depth=1 tree goes from 90.5% to only 91.8% when given 5× more data (+1.3%). A single question (stump) is the wrong model form — no amount of data teaches it to ask more questions. More data is useless against high bias.
2 — High variance does improve with more data, but slowly. The unlimited tree goes from 84.3% to 93.9% (+9.6%) with more data. More data helps variance because it forces the model to generalize — there are now more examples to match any region of input space. But the gap (train=100%, val=93.9%) persists. Regularization is needed alongside more data.
3 — The gap is your variance estimate. Depth=1 gap: 0.9% — almost no variance, just bias. Unlimited gap: 6.1% — substantial variance. This gap is what regularization targets. Depth=4 gap: 2.0% — acceptable. This is the sweet spot where both problems are managed.
06 — Every Model You've Seen, Explained Through This Lens
07 — How to Fix Each Problem: The Mechanic's Toolkit
Add features — more input signals give the model more to work with.
Add polynomial / interaction features — let a linear model approximate curves by adding x², x³, x₁·x₂ as new features.
Use a more complex model class — linear → tree → ensemble → neural network, in order of expressiveness.
Reduce regularization — regularization intentionally adds bias to control variance; reducing it gives the model more freedom.
Train longer — for neural networks, early stopping might be cutting training before the model converges.
Do NOT: collect more data. More data cannot teach a simple model to be more expressive.
Collect more training data — more examples force the model to generalize; noise cancels out across more samples.
Add regularization — L1/L2 penalize large weights; Dropout randomly removes neurons during training. Both force the model to find simpler, more robust patterns. (Day 10 covers this in depth.)
Simplify the model — reduce max_depth, reduce neurons/layers, reduce max_features.
Use ensemble methods — averaging many high-variance models reduces variance without increasing bias. This is mathematically proven: Var(average of N models) = Var(single model) / N.
Data augmentation — artificially expand the training set by transforming existing examples (flip images, add noise, paraphrase text).
Do NOT: make the model more complex — this will widen the train/test gap further.
08 — The Deep Learning Exception: Double Descent
Classical bias-variance theory predicts a U-shaped error curve: error decreases as model complexity increases (bias falling), then rises again as variance takes over. This holds for classical ML models.
But something strange happens with very large neural networks — and it's one of the most active research areas in modern ML theory.
As neural network capacity grows, the classical U-shaped error curve appears — error rises as the model becomes large enough to interpolate (perfectly memorize) all training data. This is the interpolation threshold.
But then something unexpected happens: if you keep adding capacity beyond the interpolation threshold, test error starts falling again — and often falls below the classical minimum. This is the double descent curve. Massively overparameterized models generalize surprisingly well.
Why? The leading theory: when a model has far more parameters than training examples, there are infinitely many ways to memorize the training data. Gradient descent finds the simplest (minimum-norm) solution — which tends to be smoother and more generalizable than the complex solution a smaller model would be forced to use.
GPT-4 has hundreds of billions of parameters. Classical theory predicts catastrophic overfitting. Instead it generalizes across languages, domains, and tasks it never saw. The bias-variance tradeoff hasn't disappeared — it's become more complex. Understanding the classical tradeoff is still essential: it explains everything up to the point where scale changes the rules.
09 — A Thought to Sit With
In medicine, a highly biased diagnostic tool misses cases consistently — it has the wrong model of disease. A highly variable one gives different results for the same patient depending on who runs it. But high precision (low variance) on the wrong diagnosis (high bias) can be the most dangerous outcome — it gives false confidence in a consistently wrong answer.
In safety-critical AI — medical diagnosis, autonomous vehicles, legal sentencing — which failure mode is more dangerous: consistent wrongness or inconsistent results? Does the answer change depending on whether the system operates alone or as a second opinion alongside a human expert?
10 — Coming Up Next
Tomorrow — Day 10: Regularization. Now that you know variance is the enemy in most practical ML problems, Day 10 covers the main weapons against it: L1 (Lasso), L2 (Ridge), Elastic Net, Dropout, and Early Stopping. Each one adds a specific kind of controlled bias to buy back variance. You'll understand exactly which to reach for — and why.
11 — Go Deeper
▶ StatQuest: Bias and Variance, Clearly Explained
YouTube · 6 min · The clearest visual walkthrough of the dartboard analogy and error decomposition — watch this immediately after reading.
▶ StatQuest: Learning Curves & Diagnosing Models
YouTube · Best companion to the code section — how to interpret learning curves visually and decide which fix to apply.
▶ Coursera: Deep Learning Specialization — Andrew Ng
5-course specialization · Course 2 covers bias and variance in the deep learning context, including how to systematically diagnose and fix problems in neural networks. The single best structured resource for this topic. Audit free.
- Total Error = Bias² + Variance + Irreducible Noise. You minimize the first two; the noise floor is fixed.
- Bias — error from wrong structural assumptions. Consistent, systematic error even on training data. Also called underfitting.
- Variance — error from sensitivity to training data noise. Model memorizes instead of generalizing. Also called overfitting.
- Irreducible noise — error in the data itself. No model can beat this floor.
- Diagnosis: high train & test error, small gap → high bias. Low train error, high test error, large gap → high variance.
- Fix bias: more complex model, more features, less regularization. More data does NOT help.
- Fix variance: regularization, simpler model, more data, ensembles. More complexity makes it worse.
- More data reduces variance (smooths out noise). Never reduces bias (can't change model form).
- Ensembles reduce variance by averaging — Var(average of N models) = Var(single model)/N. Bias stays the same.
- Double descent: at massive scale (LLMs), the classical tradeoff curve bends — extremely overparameterized models can achieve both low bias and low variance. The tradeoff is real but more complex in the deep learning regime.
- What is bias in machine learning?
Bias is error from wrong structural assumptions in the model. A high-bias model is too simple — it makes systematic errors even on training data because its form cannot represent the true pattern. Also called underfitting. Example: linear regression on a nonlinear relationship. - What is variance in machine learning?
Variance is error from sensitivity to training data noise. A high-variance model memorizes training data quirks instead of learning the underlying pattern — it overfits. Retrain on a different random sample and you get a noticeably different model. Example: unlimited decision tree. - How do you diagnose overfitting vs underfitting from training/test error?
High training AND test error, small gap → high bias (underfitting). Low training error but high test error, large gap → high variance (overfitting). Both low, small gap → good generalization. The gap itself is your variance estimate. - Name three ways to reduce variance (overfitting).
Regularization (L1/L2/Dropout adds controlled bias to reduce variance); simplify the model (reduce max_depth, fewer parameters); collect more training data (different datasets look more similar with more examples); use ensemble methods (averaging N models reduces variance by factor of N). - What is the bias-variance tradeoff?
Total Error = Bias² + Variance + Irreducible Noise. Reducing bias (more complex model) typically increases variance, and vice versa. The goal is the complexity level where Bias² + Variance is minimized. Ensembles partially escape this tradeoff. Very large neural networks (double descent) escape it further — but understanding the classical tradeoff is essential for diagnosing any ML model.
No attachments for this topic yet.