Bagging & Random Forests:
The Wisdom of Crowds, Formalized

It's 1906, and statistician Francis Galton is at a county fair in Plymouth.
Eight hundred people have just guessed the weight of a live ox.
Each individual guess is flawed — farmers, merchants, random passersby.
The median of all 800 guesses: 1,207 pounds.
The actual weight of the ox: 1,198 pounds.

The crowd was wrong by less than 1%. No individual came close.
This is why Random Forest is one of the most reliable models ever built.

01 — The Problem With One Tree

From Day 9 you know the central tension: decision trees sit at one extreme of the bias-variance tradeoff. A deep, unconstrained tree has very low bias — it can model any pattern — but extremely high variance. Train the same tree architecture on two slightly different datasets and you get two completely different trees. The model is fragile by design.

The question that created Random Forests is elegant in its simplicity: what if instead of training one perfect tree, you trained hundreds of deliberately imperfect ones and made them vote? Each tree would make different errors. When you average their predictions, the errors cancel out and the signal survives. This is not a heuristic — it is mathematically guaranteed, and we'll prove it in §03.

This idea — training multiple models on different random subsets of data and aggregating their predictions — is called bagging (bootstrap aggregating). Random Forests add a second layer of randomness on top that makes the technique dramatically more powerful.

02 — How It's Built: The 5-Step Process

Before the math, let's see the full mechanism. Every Random Forest follows the same construction, step by step:

03 — The Math: Why Averaging Works

This is the proof. Not hand-waving — the actual reason ensembles beat individuals, stated precisely enough to be convincing.

The key condition is uncorrelated errors. If all trees made identical errors, averaging would do nothing — you'd divide noise by B but it would still be the same noise. This is why the two randomisation steps matter:

04 — Feature Importance: What the Forest Knows About Your Data

Every time a tree makes a split on feature X, it reduces the impurity (Gini or entropy) of the node by some amount. Random Forests track this reduction across every split, across every tree, and average it per feature. The result is a robust importance score telling you which features are actually doing the predictive work.

05 — From Scratch: Understanding Bootstrap Sampling

Before sklearn abstracts everything, let's see bootstrap sampling and voting built manually. This makes the mechanism visible — you can watch the diversity being created:

import numpy as np
from collections import Counter

np.random.seed(42)
N = 10  # 10 training examples (indices 0–9)

print("=== Bootstrap Sampling Demo ===")
for i in range(5):
    sample = np.random.choice(N, size=N, replace=True)
    oob    = sorted(set(range(N)) - set(sample))
    counts = Counter(sample)
    print(f"Bootstrap {i+1}: {sorted(sample)}  |  OOB: {oob}")

print("\n=== Voting Demo: 7 Trees, 4 Predictions Each ===")
# Each row = one tree's predictions on 4 test samples
tree_preds = np.array([
    [1, 0, 1, 1],  # Tree 1
    [1, 1, 0, 1],  # Tree 2
    [0, 1, 1, 1],  # Tree 3
    [1, 0, 1, 0],  # Tree 4
    [1, 1, 1, 1],  # Tree 5
    [1, 0, 0, 1],  # Tree 6
    [0, 1, 1, 1],  # Tree 7
])
true_labels = np.array([1, 1, 1, 1])

vote_sum    = tree_preds.sum(axis=0)  # votes for class 1
forest_pred = (vote_sum > len(tree_preds) / 2).astype(int)

for j in range(4):
    print(f"Sample {j+1}: votes={vote_sum[j]}/7  forest={forest_pred[j]}  true={true_labels[j]}")
print(f"Forest accuracy: {(forest_pred == true_labels).mean():.0%}")
print(f"Best single tree: {max((tree_preds[i]==true_labels).mean() for i in range(7)):.0%}")

06 — Full Code: Single Tree vs Random Forest vs OOB

Now the real demonstration — the same dataset, three comparisons. Watch the numbers tell the story:

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier

data = load_breast_cancer()
X, y = data.data, data.target
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, random_state=42)

# ── Single Decision Tree ──────────────────────────────────────────
single_tree = DecisionTreeClassifier(random_state=42)
single_tree.fit(X_tr, y_tr)
tree_cv = cross_val_score(single_tree, X, y, cv=10)

# ── Random Forest: 300 trees ──────────────────────────────────────
rf = RandomForestClassifier(
    n_estimators=300,        # 300 trees in the ensemble
    max_features='sqrt',     # √30 ≈ 5 features per split
    max_depth=None,           # fully grown trees (low bias)
    oob_score=True,          # free validation from ~37% left-out examples
    n_jobs=-1,               # use all CPU cores (trees train in parallel)
    random_state=42
)
rf.fit(X_tr, y_tr)
rf_cv = cross_val_score(rf, X, y, cv=10)

# ── Results ───────────────────────────────────────────────────────
print("Model            Train     Test    CV Mean  CV Std  Stable?")
print("-" * 65)
print(f"Single Tree      {single_tree.score(X_tr,y_tr):.3f}    {single_tree.score(X_te,y_te):.3f}   {tree_cv.mean():.3f}    {tree_cv.std():.3f}    {'No' if tree_cv.std()>0.03 else 'Yes'}")
print(f"Random Forest    {rf.score(X_tr,y_tr):.3f}    {rf.score(X_te,y_te):.3f}   {rf_cv.mean():.3f}    {rf_cv.std():.3f}    {'No' if rf_cv.std()>0.03 else 'Yes'}")
print(f"OOB Score (free validation):  {rf.oob_score_:.3f}")
print(f"OOB vs CV gap:                {abs(rf.oob_score_ - rf_cv.mean()):.3f}")

# ── Feature Importance ────────────────────────────────────────────
print("\n── Top 5 Most Important Features ──")
importances = rf.feature_importances_
top5_idx = np.argsort(importances)[::-1][:5]
for rank, idx in enumerate(top5_idx, 1):
    bar = "█" * int(importances[idx] * 100)
    print(f"  #{rank}  {data.feature_names[idx]:<30} {importances[idx]:.3f}  {bar}")

# ── Confidence via vote margin ─────────────────────────────────────
print("\n── Prediction Confidence (first 5 test samples) ──")
proba = rf.predict_proba(X_te)
for i in range(5):
    confidence = max(proba[i])
    label = data.target_names[rf.predict(X_te[i:i+1])[0]]
    true  = data.target_names[y_te[i]]
    match = "✓" if label == true else "✗"
    print(f"  Sample {i+1}: predicted={label:<12} confidence={confidence:.1%}  true={true} {match}")

07 — Hyperparameters That Actually Matter

08 — When Random Forest Wins (and When It Doesn't)

09 — A Thought to Sit With

10 — Coming Up Next

Day 12 — Boosting: AdaBoost & Gradient Boosting. Bagging trains trees in parallel and averages them. Boosting does the opposite: trees are built sequentially, each one focusing on the mistakes the previous one made. A fundamentally different philosophy — and the foundation of XGBoost, the algorithm that has won more Kaggle competitions than any other.

11 — Go Deeper

▶ StatQuest: Random Forests, Clearly Explained
YouTube · 9 min · Best visual walkthrough of bootstrap sampling and feature randomness — watch this immediately after reading §02.

▶ StatQuest: Random Forest Part 2 — Missing Data & Clustering
YouTube · 12 min · Advanced uses including proximity matrices for missing data imputation and unsupervised clustering.

▶ Random Forests — Leo Breiman (2001)
The original paper. Surprisingly readable. Sections 1–3 cover everything in this post with the full statistical proofs. Breiman invented Random Forests — reading the source is worthwhile.