A doctor looks at a patient's blood test results.
Cholesterol: 240. Blood pressure: 150/95. Age: 58. Smoker: yes.
She doesn't say: "Your heart disease score is 3.7."
She says: "There's an 80% chance you'll have a cardiac event in the next 10 years."
That's not linear regression. That's logistic regression.
01 — The Problem With Using Linear Regression for Classification
Suppose you want to predict: will this email be spam (1) or not spam (0)? You could try linear regression. The problem: linear regression outputs any number — it might predict 2.3 for a very spammy email, or -0.8 for a clean one. These are not probabilities. They don't even make sense as outputs for a yes/no question.
We need a model that outputs a number between 0 and 1 — interpretable as a probability. Logistic regression does exactly this by wrapping the linear model inside one clever function.
Think of a dimmer switch. Linear regression is a regular switch that can go from -100 to +100. You need something that, no matter how extreme the input, always outputs between 0 and 1. The sigmoid function is that dimmer — it squishes any input into the range (0, 1), smoothly, without discontinuity.
02 — The Sigmoid Function
The sigmoid (also written σ, the Greek letter "sigma") takes any real number and squashes it into the range (0, 1). The formula is:
σ(z) = 1 / (1 + e−z)
e is Euler's number — approximately 2.718, the base of natural logarithms. You don't need to memorize it. What matters is the behavior: as z grows very large, e−z shrinks to zero and σ(z) approaches 1. As z becomes very negative, e−z grows enormous and σ(z) approaches 0. At exactly z = 0, σ(0) = 0.5.
The full logistic regression model first computes the logit — the raw linear combination:
z = w1x1 + w2x2 + ... + wnxn + b
Then applies sigmoid to get the probability: p = σ(z). That probability is the final output — the model's confidence that the input belongs to class 1.
Suppose a trained model learns w = 2.1 and b = −7.3 for tumor size. A patient has tumor size x = 3.8cm:
z = 2.1 × 3.8 − 7.3 = 7.98 − 7.3 = 0.68
p = σ(0.68) = 1 / (1 + e−0.68) ≈ 0.664
The model outputs 66.4% — meaning it believes there is a 66.4% probability that this tumor is malignant. Since 0.664 > 0.5, it predicts class 1 (malignant).
03 — The Decision Boundary
Once the model outputs a probability, we need a rule to make a final yes/no decision. The default: predict class 1 if p ≥ 0.5, class 0 if p < 0.5. The decision boundary is the precise line in input space where p = exactly 0.5 — the dividing line between predictions.
Since p = 0.5 when σ(z) = 0.5, and σ(z) = 0.5 when z = 0, the decision boundary is simply the set of inputs where w1x1 + ... + b = 0. That's a linear equation — which means the boundary is always a straight line in 2D, a flat plane in 3D, and a hyperplane in higher dimensions.
If your two classes can't be separated by a straight line — if they're arranged in circles, spirals, or overlapping blobs — logistic regression will always draw a straight boundary and do its best, but it will never perfectly separate them. You can see this happening when training accuracy plateaus at a value clearly below 100% even with many epochs.
Fixes: Add polynomial features (x², x₁·x₂) to your input — this curves the boundary in the original space. Or switch to a non-linear model like a decision tree, SVM with a radial kernel, or a neural network.
The 0.5 threshold is not sacred. In fraud detection, you might lower it to 0.3 — catching more real fraud at the cost of more false alarms. In medical screening, you might lower it further to ensure almost no cases are missed, accepting a high false positive rate. The model gives you the probability; you choose where to draw the line based on the real-world cost of each type of mistake.
04 — How It Trains: Log-Loss (Cross-Entropy)
Logistic regression is trained with cross-entropy loss, also called log-loss. For a single example with true label y (either 0 or 1) and predicted probability p, the loss is:
Loss = −[ y · log(p) + (1 − y) · log(1 − p) ]
Let's unpack this. The formula has two branches — only one activates per example:
p = 0.99 → loss ≈ 0.01 ✓
tiny — model was confident and correct
p = 0.50 → loss ≈ 0.69 △
uncertain — moderate penalty
p = 0.01 → loss ≈ 4.60 ✗
enormous — model was confident and completely wrong
p = 0.01 → loss ≈ 0.01 ✓
tiny — model was confident and correct
p = 0.50 → loss ≈ 0.69 △
uncertain — moderate penalty
p = 0.99 → loss ≈ 4.60 ✗
enormous — high confidence in the wrong direction is catastrophic
The reason we use logarithm rather than a simpler function: log grows very steeply near zero. A confident wrong prediction (p ≈ 0 when y = 1) creates a massive loss — and therefore a large gradient — driving the model to correct the mistake forcefully. MSE near the same point would give a loss of only ~1.0, providing a weak gradient signal. Cross-entropy makes training much faster and more stable for classification.
05 — From Scratch + sklearn
The from-scratch version makes every step explicit — sigmoid, cross-entropy gradient, and weight update. The sklearn version does the same thing with one line, using a more robust numerical solver internally.
import numpy as np
# Sigmoid: squashes any number into (0, 1)
def sigmoid(z): return 1 / (1 + np.exp(-z))
# Toy binary dataset: tumor size (cm) → malignant? (0=no, 1=yes)
X = np.array([1.5, 2.0, 2.5, 3.0, 3.5, 4.0, 4.5, 5.0])
y = np.array([ 0, 0, 0, 0, 1, 1, 1, 1])
# ── From scratch: gradient descent on cross-entropy loss ─────────────
w, b, lr = 0.0, 0.0, 0.1
for _ in range(3000):
p = sigmoid(w * X + b) # predicted probabilities
w -= lr * np.mean((p - y) * X) # gradient of cross-entropy w.r.t. w
b -= lr * np.mean(p - y) # gradient w.r.t. b
print(f"Learned: w={w:.2f}, b={b:.2f}")
prob_scratch = sigmoid(w * 3.8 + b)
print(f"P(malignant | size=3.8cm) = {prob_scratch:.1%}")
# ── sklearn equivalent ────────────────────────────────────────────────
from sklearn.linear_model import LogisticRegression
import warnings; warnings.filterwarnings('ignore')
clf = LogisticRegression().fit(X.reshape(-1, 1), y)
prob_sklearn = clf.predict_proba([[3.8]])[0, 1]
print(f"sklearn P(malignant | size=3.8cm) = {prob_sklearn:.1%}")
print(f"sklearn Decision for 3.8cm: {'Malignant' if prob_sklearn >= 0.5 else 'Benign'}")w = 2.97 — each additional centimetre of tumor size increases the logit by 2.97. Because this feeds into a sigmoid, the effect on probability is non-linear: it matters most near the decision boundary (z ≈ 0), and has diminishing impact far from it.
b = −9.57 — the baseline logit with zero size. A very negative number, which makes sense: a tumor of size 0cm should have near-zero malignancy probability.
87–88% — the small difference between scratch and sklearn is normal. sklearn uses a more numerically robust solver (LBFGS), not plain gradient descent. Both are correct.
06 — Regularization & When to Use It
Like linear regression, logistic regression can overfit when you have many features and limited data. Regularization adds a penalty for large weights, keeping the model simpler. sklearn's LogisticRegression() applies L2 regularization by default — controlled by the C parameter.
penalty='l2')penalty='l1')The final layer of BERT, GPT, and most classification neural networks is a logistic regression head — a linear layer followed by sigmoid (for binary) or softmax (for multi-class). When you fine-tune a pretrained model for sentiment analysis, you are essentially training a logistic regression on top of the model's learned representations. The 70-year-old algorithm is the last piece of the most modern architectures.
07 — A Thought to Sit With
Logistic regression outputs a probability — say, 73% chance of fraud. But the final decision (flag or don't flag) requires a threshold, typically 50%. Lowering it to 30% catches more real fraud — but also flags more legitimate transactions, frustrating honest customers.
Who should decide where that threshold sits — the data scientist, the business, or the regulator? And whose interests might conflict when each makes that choice?
08 — Coming Up Next
Tomorrow — Day 6: K-Nearest Neighbors. We abandon equations entirely. KNN makes predictions by asking "what do my closest neighbors predict?" — and in doing so, teaches something profound about the nature of similarity and distance in machine learning.
09 — Go Deeper
▶ StatQuest: Logistic Regression, Clearly Explained
YouTube · 19 min · The clearest visual explanation of odds, log-odds, and why sigmoid is the natural choice. Watch this directly after reading.
▶ Kaggle: Intro to Machine Learning
Free interactive course · Apply logistic regression to real classification datasets in your browser. The fastest way to go from reading to hands-on practice.
- What it does — Predicts the probability that an input belongs to a binary class (yes/no, spam/ham, malignant/benign).
- The logit (z) — Raw linear score: w·x + b. Unconstrained — any real number.
- Sigmoid σ(z) — Squashes logit into (0, 1). σ(0) = 0.5. Large positive → near 1. Large negative → near 0.
- Decision boundary — Where z = 0 and σ(z) = 0.5. Always a straight line or hyperplane.
- Cross-entropy loss — −[y·log(p) + (1−y)·log(1−p)]. Penalizes confident wrong predictions enormously.
- Why not MSE? — Cross-entropy creates much stronger gradients for confident mistakes, making training faster and more stable.
- Threshold — Default 0.5, but adjustable to control the tradeoff between false positives and false negatives.
- Regularization — C controls strength (smaller = stronger). L1 removes irrelevant features; L2 shrinks all weights. L2 is the default.
- In modern AI — The classification head of BERT, GPT, and almost every neural network is a logistic regression (or its multi-class generalization, softmax).
- Why can't we use linear regression for classification?
Linear regression outputs unbounded values that don't represent valid probabilities. It also uses MSE, which produces weak gradient signals for binary targets — cross-entropy is far more effective because it heavily penalizes confident wrong predictions. - What is the sigmoid function and why do we use it?
σ(z) = 1/(1+e−z). It maps any real number to (0, 1), making the output a valid probability. It is smooth and differentiable everywhere, enabling gradient descent. It is also the natural inverse of the log-odds transformation. - What loss function does logistic regression use and why not MSE?
Cross-entropy: −[y·log(p) + (1−y)·log(1−p)]. It penalizes confident wrong predictions with a very large loss (log approaches infinity near 0), creating strong gradients. MSE gives approximately 1.0 loss for the same confident mistake — a much weaker corrective signal. - What is the decision boundary in logistic regression?
The set of inputs where σ(z) = 0.5, equivalently where z = 0. Since z = w·x + b is linear, the boundary is always a line, plane, or hyperplane. It cannot represent curved class boundaries without polynomial feature engineering. - How does the C parameter in sklearn's LogisticRegression work?
C is the inverse of regularization strength. Small C = strong regularization (simpler model, less overfitting). Large C = weak regularization (fits training data closely, risk of overfitting). Default C=1 is a good starting point.
No attachments for this topic yet.