How Models Actually Learn:
Loss, Gradient Descent & Optimization

You're blindfolded on a hilly landscape. Your goal: reach the lowest valley.
You can't see anything. But you can feel the ground beneath your feet.
You take a small step in whichever direction feels most downhill.
Then another. Then another.

Eventually — step by step — you find the bottom.
That is gradient descent. That is how every ML model learns.

01 — What Does "Learning" Actually Mean?

When we say a model "learns," we mean something very specific: it adjusts its internal numbers — called parameters or weights — until its predictions become as accurate as possible on the training data.

Before training, these weights are random. The model makes terrible predictions. After training, the weights have been tuned so that predictions are good. The process of tuning them is called optimization, and it is driven entirely by a loss function.

Think of it like tuning a guitar. Each string (weight) starts out of tune. You tighten or loosen each one — just a little at a time — guided by the sound it makes (the loss). You stop when the guitar sounds right (loss is low). The model does exactly this, but with millions of strings tuned simultaneously, thousands of times.

02 — The Loss Function: Measuring How Wrong You Are

A loss function is a mathematical formula that takes your model's prediction and the correct answer, and returns a single number: the error. Lower loss = better predictions. The goal of training is to minimize this number.

Different problems use different loss functions. The two most important ones to know:

03 — Gradient Descent: The Optimization Engine

Once we can measure error (loss), we need a way to reduce it. That mechanism is gradient descent.

The gradient is the slope of the loss function at the model's current position. It answers the question: "If I increase this weight slightly, does the loss go up or down — and by how much?" The gradient always points uphill (toward higher loss). So we go in the opposite direction — downhill — to reduce the loss. This is exactly the blindfolded hiker from the opening: you feel the slope, and step downhill.

04 — The Learning Rate: The Most Important Number You'll Tune

The learning rate (written as α or lr) controls how large each weight update step is. It is the single most impactful hyperparameter in training — and the one engineers spend the most time tuning.

05 — Gradient Descent in Pure NumPy

Here is the complete training loop in about 15 lines of Python — no frameworks, no abstractions. Every major library (PyTorch, TensorFlow, sklearn) does exactly this internally, just faster and with automatic differentiation.

import numpy as np

# Toy dataset: learn y = 2x + 1 from noisy observations
np.random.seed(42)
X = np.random.randn(100)
y = 2 * X + 1 + np.random.randn(100) * 0.3  # true relationship + noise

# Initialize weights at zero — the model knows nothing yet
w, b = 0.0, 0.0   # weight and bias
lr    = 0.01       # learning rate

for epoch in range(1000):
    y_pred = w * X + b                    # Step 1: forward pass
    loss   = np.mean((y_pred - y) ** 2)  # Step 2: MSE loss

    # Step 3: gradients — how does loss change if we nudge w or b?
    dw = np.mean(2 * (y_pred - y) * X)   # dLoss/dw
    db = np.mean(2 * (y_pred - y))        # dLoss/db

    # Step 4: update — move opposite to the gradient
    w -= lr * dw
    b -= lr * db

print(f"Learned: w={w:.3f}, b={b:.3f}")
# → Learned: w≈2.000, b≈1.000  ✓  (recovered the true relationship)

The model started knowing nothing (w=0, b=0). After 1000 steps of gradient descent, it recovered the exact relationship hidden in the data — y = 2x + 1. This is the essence of machine learning.

06 — How Much Data Per Update?

The training loop we've seen computes the gradient using the entire dataset before making each update. That works — but with large datasets, waiting to see every example before taking one step is painfully slow. Engineers choose how many examples to use per update, giving rise to three variants:

Beyond how many examples per update, there's also the question of how to apply the update itself. The basic rule — weight = weight − lr × gradient — gives every weight the same step size on every update. That's like telling every member of a hiking group to take exactly the same stride length, regardless of whether they're on flat ground or a steep cliff. It works, but it's not efficient.

Adam (Adaptive Moment Estimation) is smarter. Think of it as each weight keeping a personal notebook. In that notebook it records: "How have my recent updates been going? Am I consistently heading in the same direction, or am I bouncing back and forth?" If a weight has been moving steadily in the same direction — it's making confident progress — Adam lets it take bigger steps. If a weight keeps changing direction and seems uncertain — Adam slows it down and makes it more careful. The result: faster overall convergence, and far less sensitivity to the initial learning rate you pick. This is why Adam is the default starting point for almost every training project today.

07 — The Hidden Problem: Local vs Global Minima

Here's something that bothered mathematicians for a long time about gradient descent: it only follows the slope downhill from wherever it starts. It has no map. It can't see the full landscape. It just feels the ground under its feet and takes the next step.

This creates a genuine problem. Imagine you're dropped blindfolded onto a mountain range. You follow the slope downhill faithfully — step by step — and eventually reach flat ground. You take off your blindfold. You're in a valley. But looking around, you can see there's a much deeper valley on the other side of a ridge you never crossed. You found a bottom — but not the bottom.

In ML terms: the valley you found is a local minimum — a point where the loss is lower than anything nearby, so gradient descent stops there. The deepest valley in the entire landscape is the global minimum — the theoretically best solution. Gradient descent is not guaranteed to find it.

So does this mean gradient descent is broken? Not in practice. Here's why:

First, a good enough solution is usually good enough. In most real problems — predicting prices, classifying images, filtering spam — a local minimum that achieves 94% accuracy is perfectly useful, even if the global minimum might give 95%. The difference rarely justifies the cost of finding it.

Second, the mini-batch noise actually helps. Because mini-batch GD uses a random subset of data each time, its loss curve is slightly noisy — it bounces a little rather than descending perfectly smoothly. This noise can accidentally knock the model out of a shallow local minimum and into a better one. What looks like a flaw turns out to be a feature.

Third, Adam has a built-in escape mechanism. When Adam notices that a weight keeps reversing direction — a sign it may be stuck near a local minimum — it reduces that weight's step size, which ironically makes the updates more consistent and can help navigate the landscape more carefully.

08 — Coming Up Next

Tomorrow — Day 4: Linear Regression. The simplest model that uses everything you just learned. One loss function, one gradient descent loop, one straight line — and it teaches more about ML than almost any other algorithm.

09 — Go Deeper

▶ 3Blue1Brown: Gradient Descent, How Neural Networks Learn
YouTube · 21 min · The definitive visual explanation of exactly this topic. The animation of the loss landscape makes the concept click in a way words can't.

▶ 3Blue1Brown: What is Backpropagation Really Doing?
YouTube · 14 min · Builds directly on the gradient descent video — shows how the error signal travels backwards to assign blame to each weight. No calculus required to follow the intuition.

▶ Coursera: Deep Learning Specialization — Andrew Ng
5-course specialization · The gold standard for learning the math behind gradient descent, backpropagation, and optimization — explained step by step with the clarity Andrew Ng is known for. Audit for free.