Decision Trees:
The Model You Can Read Like a Book

A doctor sees a patient and thinks:
"Is the fever above 38.5°C? → Yes.
Is there a cough? → Yes.
Is the cough productive? → No.
Duration more than 5 days? → Yes."
Diagnosis: viral upper respiratory infection. Prescribe rest and fluids.

This is not a neural network. This is a decision tree —
and your doctor has been running one for every patient, every day.

01 — What a Decision Tree Is

A decision tree is a model that learns a hierarchy of if-else rules directly from data. It has three types of components:

To classify a new example, you start at the root node, answer each question, follow the appropriate branch, and keep going until you reach a leaf. The leaf's label is the prediction. That's the entire inference process — no matrix multiplication, no gradients, just a sequence of if-else checks.

Unlike every model seen so far, a trained decision tree can be written out as human-readable rules. A doctor can verify it. A judge can audit it. A regulator can approve it. This interpretability is why decision trees remain mandatory in healthcare, finance, and law — even when less interpretable models would be more accurate.

02 — How the Tree Grows: Gini Impurity & Information Gain

At each node, the algorithm searches for the best split: which feature, and which threshold, produces the purest child nodes? "Purity" means one class dominates — the opposite of a 50/50 mix. Three concepts build on each other here, so walk through them in order:

03 — The Tree-Growing Algorithm: Step by Step

The algorithm is recursive — it builds the tree by repeatedly asking "what is the best split I can make at this node?" until a stopping condition is met.

04 — The Overfitting Problem: Memorizing Noise

Left unconstrained, a decision tree will keep growing until every single training example has its own leaf — 100% training accuracy. This is the most textbook example of overfitting in all of machine learning.

05 — Pruning: Two Ways to Fight Overfitting

06 — From Scratch: A Minimal Decision Tree

Building a tree from scratch reveals the recursive structure that sklearn hides. This is the core logic in ~30 lines — no fancy optimizations, just the algorithm.

import numpy as np
from collections import Counter

def gini(y):
    """Gini impurity of a label array."""
    n = len(y)
    if n == 0: return 0
    counts = Counter(y)
    return 1 - sum((c / n) ** 2 for c in counts.values())

def best_split(X, y):
    """Find feature + threshold that gives the highest Information Gain."""
    best_gain, best_feat, best_thresh = -1, None, None
    parent_gini = gini(y)

    for feat in range(X.shape[1]):              # each feature
        thresholds = np.unique(X[:, feat])
        for t in thresholds:                    # each threshold
            left  = y[X[:, feat] <= t]
            right = y[X[:, feat] >  t]
            if len(left) == 0 or len(right) == 0: continue

            # Information Gain = parent impurity − weighted child impurity
            n = len(y)
            gain = parent_gini - (len(left)/n * gini(left) + len(right)/n * gini(right))
            if gain > best_gain:
                best_gain, best_feat, best_thresh = gain, feat, t

    return best_feat, best_thresh, best_gain

def build_tree(X, y, depth=0, max_depth=3):
    """Recursively build a decision tree. Returns a dict node."""
    # Stopping conditions
    if depth >= max_depth or len(set(y)) == 1:
        return {'leaf': Counter(y).most_common(1)[0][0]}

    feat, thresh, gain = best_split(X, y)
    if gain <= 0:   # no useful split exists
        return {'leaf': Counter(y).most_common(1)[0][0]}

    mask = X[:, feat] <= thresh
    return {
        'feature': feat, 'threshold': thresh, 'gain': round(gain, 3),
        'left' : build_tree(X[mask],  y[mask],  depth+1, max_depth),
        'right': build_tree(X[~mask], y[~mask], depth+1, max_depth),
    }

def predict_one(node, x):
    if 'leaf' in node: return node['leaf']
    if x[node['feature']] <= node['threshold']:
        return predict_one(node['left'], x)
    return predict_one(node['right'], x)

# Quick test on a toy dataset
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split

X, y = load_iris(return_X_y=True)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, random_state=42)

tree = build_tree(X_tr, y_tr, max_depth=3)
preds = [predict_one(tree, x) for x in X_te]
accuracy = np.mean(np.array(preds) == y_te)
print(f"From-scratch tree accuracy: {accuracy:.1%}")
print(f"Root split: feature {tree['feature']} ≤ {tree['threshold']} (gain={tree['gain']})")

07 — sklearn: Overfitting vs Pruned + Feature Importance

The sklearn version handles all optimizations and exposes the full comparison — overfitting vs pruning — plus feature importance scores.

from sklearn.tree import DecisionTreeClassifier, export_text
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
import pandas as pd

data = load_breast_cancer()
X, y = data.data, data.target
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, random_state=42)

# ── Unlimited tree: maximum overfitting ───────────────────────────────
t1 = DecisionTreeClassifier(random_state=42).fit(X_tr, y_tr)
print(f"Unlimited  │ train: {t1.score(X_tr,y_tr):.1%}  test: {t1.score(X_te,y_te):.1%}  depth: {t1.get_depth()}")

# ── Pre-pruned (max_depth=4): better generalization ───────────────────
t2 = DecisionTreeClassifier(max_depth=4, random_state=42).fit(X_tr, y_tr)
print(f"max_depth=4 │ train: {t2.score(X_tr,y_tr):.1%}  test: {t2.score(X_te,y_te):.1%}")

# ── Feature importance: which features matter most? ───────────────────
# Each score = total Gini reduction from all splits using this feature,
# weighted by number of samples, normalized so all scores sum to 1.0
imp = pd.Series(t2.feature_importances_, index=data.feature_names).sort_values(ascending=False)
print("\nTop 5 features by importance:")
print(imp.head(5).to_string())

# ── Human-readable rules for the first 2 levels ───────────────────────
print("\nTree rules (depth ≤ 2):")
print(export_text(t2, feature_names=list(data.feature_names), max_depth=2))

08 — Feature Importance: The Free Gift

Every trained decision tree delivers feature importance scores as a byproduct. The formula for each feature's importance is:

Importance(feature) = Σ [ (nₜ/n) × ΔGini(t) ] over all nodes t that split on this feature

Where nₜ is the number of samples at node t, n is total training samples, and ΔGini(t) is the Gini reduction from the split. Features used at high-sample, high-impurity nodes (early in the tree) get higher scores. All scores are normalized to sum to 1.0.

09 — A Thought to Sit With

10 — Coming Up Next

Tomorrow — Day 9: Bias, Variance & the Tradeoff. The most important concept in all of machine learning. Every model, every hyperparameter, every design choice moves a single dial between two failure modes. Understanding this one idea explains almost every phenomenon you've seen so far — including why the unlimited tree failed and the pruned tree succeeded.

11 — Go Deeper

▶ StatQuest: Decision Trees, Clearly Explained
YouTube · 18 min · The best visual explanation of Gini impurity, Information Gain, and how a tree grows step by step.

▶ Kaggle: Intro to Machine Learning
Free interactive course · Decision trees are the first model in this course — apply them to real datasets in your browser with immediate feedback.

▶ Coursera: Deep Learning Specialization — Andrew Ng
5-course specialization · Covers bias-variance tradeoff rigorously — the concept that explains every overfitting and underfitting phenomenon you've seen across the series. Audit free.