Day 4 — Linear Regression | 30-Day AI Ramadan Series

In 1886, Francis Galton studied the heights of parents and their children.
He noticed something: tall parents tend to have tall children — but shorter
than themselves. Short parents tend to have taller children — but shorter
than average. Heights "regress toward the mean."

He drew a line through his data. That line became the most important tool
in the history of data analysis.

01 — What Linear Regression Does

Linear regression answers one question: given some input features, what number do I predict as output? It finds the best straight line (or hyperplane in multiple dimensions) through your data that minimizes prediction error.

Examples: predict house price from size + bedrooms. Predict a student's exam score from study hours. Predict tomorrow's temperature from today's. Any time your output is a continuous number, linear regression is your first tool to reach for.

📏 Analogy

Imagine plotting every house sale in a city as a dot on graph paper — horizontal axis is size in m², vertical axis is price. Linear regression draws the single straight line that comes closest to all those dots simultaneously. Once you have that line, predicting the price of a new house is as simple as finding its size on the x-axis and reading off the y-axis.

02 — The Math (Made Friendly)

Before the formula — what is the model actually doing? It makes a weighted sum of your inputs, then adds a baseline. "Weighted sum" means: multiply each feature by how important it is, then add everything up. That's it.

Concretely — predicting an exam score from hours studied:

predicted score = (5.2 × hours_studied) + 30

The 5.2 is the weight — each extra hour of study adds 5.2 points. The 30 is the bias — the baseline score someone gets with zero hours studied. Training is the process of finding these numbers automatically from data.

In general notation with n features, the formula is:

ŷ = w₁x₁ + w₂x₂ + ... + w_nx_n + b

Where ŷ (y-hat) is the prediction, x values are your input features, w values are the learned weights (one per feature), and b is the bias.

Training finds the weights and bias that minimize MSE (Mean Squared Error) — the average of squared differences between predictions and true values. Squaring does two things: it makes all errors positive (a $10k under-prediction and $10k over-prediction don't cancel out), and it punishes large mistakes much more than small ones.

There are two ways to find these optimal values:

OLS — Ordinary Least Squares

A direct mathematical formula: w = (XᵀX)⁻¹Xᵀy. Computes the exact best weights in one step — no iterations, no learning rate to tune. Fast and exact for datasets that fit in memory. This is what sklearn's LinearRegression() uses internally.

Gradient Descent

The iterative approach from Day 3 — adjust weights step by step, guided by the loss. Slower to converge but scales to massive datasets where OLS would require inverting an enormous matrix. Used in neural networks and large-scale systems.

Predicts

A continuous number — price, temperature, score, demand

Loss function

MSE — average of (prediction − truth)²

What it learns

One weight per feature + one bias term

Key metric

R² — how much of the variance is explained by the model

03 — What R² Tells You

R² (R-squared, or the coefficient of determination) answers one question: how much better is my model than just predicting the average every time?

If the only thing you knew about exam scores was their average — say 65 — and you predicted 65 for every student, that's your dumb baseline. R² measures how much your model beats that baseline, as a proportion from 0 to 1.

R² = 1.0

Perfect. Every prediction is exactly right. The model explains 100% of why the target varies across examples.

R² = 0.85

Good. Your model explains 85% of the variation in the target. The remaining 15% is unexplained — perhaps unmeasured features (renovation quality, neighborhood reputation) or genuine randomness.

R² = 0.0

Useless. The model is exactly as good as predicting the mean every time. It learned nothing from the features.

R² < 0

Actively harmful. The model is worse than just guessing the mean. This happens when the model fits the wrong pattern entirely — usually a sign of severe underfitting or using the model on data it was never designed for.

⚠️ Context always matters

R² = 0.7 is outstanding when predicting stock prices (highly random by nature), but would be considered poor for predicting the boiling point of water (a nearly deterministic physical relationship). Never interpret R² in isolation — compare it against the baseline difficulty of your problem and the standards in your domain.

04 — Code: From Scratch vs sklearn

Two approaches to fit the same model. The from-scratch version uses gradient descent so you can see exactly what's happening inside. The sklearn version uses OLS — the direct formula, faster and exact. Both converge to the same answer.

import numpy as np

# Dataset: hours studied → exam score
X = np.array([1,2,3,4,5,6,7,8], dtype=float)
y = np.array([45,50,58,61,68,74,79,84], dtype=float)

# ── From scratch: gradient descent ──────────────────────────────────
w, b, lr = 0.0, 0.0, 0.01
for _ in range(5000):
    pred = w * X + b
    dw = np.mean(2 * (pred - y) * X)   # gradient of loss w.r.t. w
    db = np.mean(2 * (pred - y))        # gradient of loss w.r.t. b
    w -= lr * dw
    b -= lr * db
print(f"Scratch → w={w:.2f}, b={b:.2f}")

# ── sklearn: OLS (exact formula, one step) ───────────────────────────
from sklearn.linear_model import LinearRegression
model = LinearRegression().fit(X.reshape(-1,1), y)
print(f"sklearn → w={model.coef_[0]:.2f}, b={model.intercept_:.2f}")
print(f"R²      = {model.score(X.reshape(-1,1), y):.3f}")
print(f"9 hrs → {model.predict([[9]])[0]:.1f} predicted score")

→ output

Scratch → w=5.24, b=39.30 sklearn → w=5.24, b=39.30 R² = 0.996 9 hrs → 86.5 predicted score

💡 Reading the numbers

w = 5.24 — each extra hour of study adds ~5.24 points to the predicted score. This is the slope of the line: how steeply the prediction rises as the input increases.

b = 39.30 — a student who studied zero hours is predicted to score ~39 points. This is the y-intercept: the baseline the model starts from before accounting for study time.

R² = 0.996 — the model explains 99.6% of the variation in scores. Near-perfect — which makes sense, because this dataset was designed to be almost perfectly linear.

9 hrs → 86.5 — for a new student the model has never seen, studying 9 hours, the predicted score is 86.5.

05 — When Linear Regression Breaks

Linear regression makes specific assumptions. When the data violates them, the model misbehaves — sometimes silently. Here are the four most common failure modes and what to do about each.

✗

The relationship isn't actually linear. A line assumes adding one bedroom always adds the same value — regardless of whether it's going from 1 to 2 bedrooms in a studio, or from 5 to 6 in a mansion. Real data often curves. The model will systematically underpredict at the extremes and overpredict in the middle.

Fix: plot your residuals (prediction errors) against each input feature. A curved pattern in the residual plot is the telltale sign. Add polynomial features (x², x³) to capture the curve, or switch to a non-linear model like a decision tree.

✗

Outliers dominate the fit. Because MSE squares errors, one extreme data point — an absurdly priced mansion, a data entry error — can have enormous gravitational pull on the line, dragging it away from the majority of your data to chase one bad point.

Fix: always visualize your data before training. If outliers are errors, correct or remove them. If they're real but exceptional, consider using MAE (Mean Absolute Error) as your loss instead — it doesn't square errors, so outliers have proportionally less influence.

✗

Features are highly correlated (multicollinearity). If you include both "size in m²" and "size in ft²" as separate features, they carry identical information. The model can't separate their contributions — small random fluctuations in the data can cause their weights to vary wildly between training runs.

Fix: check your correlation matrix before training. If two features have a correlation above ~0.9, drop one. For a more rigorous test, compute the Variance Inflation Factor (VIF) — values above 5–10 indicate a problem. Ridge regression can also partially mitigate multicollinearity by shrinking unstable weights.

✗

You're predicting a category, not a number. Linear regression outputs any real number — including negatives and values above 1. You can't use it to predict probabilities (which must stay between 0 and 1) or class labels.

Fix: use logistic regression, which wraps the linear output in a sigmoid function to squash it into [0, 1]. That's exactly what Day 5 covers.

🔗 Connection to Modern AI

The output layer of almost every neural network is a linear layer. The classification head of a vision model, the projection layer of a transformer, the value head in reinforcement learning — all linear transformations. Linear regression isn't a historical relic. It's the atomic unit that everything else is built from.

06 — A Thought to Sit With

💬 For You to Consider

MSE treats all errors equally by size — a $10,000 mistake on a $100,000 house and a $10,000 mistake on a $10,000,000 mansion are penalized identically, even though one is a 10% error and the other is 0.1%.

When might minimizing absolute error be the wrong objective? Who is harmed when the loss function doesn't reflect real-world costs — and what would you optimize instead?

07 — Coming Up Next

Tomorrow — Day 5: Logistic Regression. Linear regression's sibling for classification. We add one elegant twist — a sigmoid function — and suddenly our model outputs probabilities. It's 70 years old and still runs in production at Google and Amazon.

08 — Go Deeper

▶ StatQuest: Linear Regression, Clearly Explained
YouTube · 27 min · The clearest visual walkthrough of OLS, R², and the assumptions that exists. Watch the residual plots section especially.

▶ Kaggle: Intro to Machine Learning
Free interactive course · Write and run real regression code on real datasets directly in your browser — the fastest path from reading to doing.

▶ Coursera: Deep Learning Specialization — Andrew Ng
5-course specialization · Covers the mathematics of regression, gradient descent, and optimization with exceptional clarity. Audit free. The best structured path from intuition to rigorous understanding.

🧠 Check Your Understanding

1. A trained model gives w = 5.24 and b = 39.3. What do these numbers actually mean?

w = 5.24 means each additional unit of the input (e.g. one more hour of study) adds 5.24 units to the prediction. It is the slope — how steeply the prediction rises as the input increases. b = 39.3 is the bias — the predicted output when all inputs are zero. For the exam score example, it means a student who studied zero hours is predicted to score ~39 points. It is the y-intercept of the line.

2. Your model scores R² = 0.0. What does this mean — and is R² = −0.2 better or worse?

R² = 0.0 means the model is exactly as useful as predicting the average of all training labels every time — it has learned nothing from the features whatsoever. R² = −0.2 is worse: the model makes predictions that are less accurate than the simple mean baseline. A negative R² typically means the model has learned the wrong pattern — usually a sign of a badly misspecified model or evaluating on data it was never designed for.

3. What is OLS and how does it differ from gradient descent?

OLS (Ordinary Least Squares) computes the exact optimal weights in a single mathematical step using the formula w = (XᵀX)⁻¹Xᵀy. No iterations, no learning rate. Fast and exact for datasets that fit in memory — which is why sklearn's LinearRegression() uses it. Gradient descent is iterative: it adjusts weights step by step guided by the loss gradient. Slower to converge but scales to very large datasets where inverting a large matrix would be computationally prohibitive. For small-to-medium regression problems, always prefer OLS.

4. Why does MSE square the error instead of just using the raw difference?

Without squaring, positive and negative errors cancel each other out — a model that overshoots by $50k on one house and undershoots by $50k on another has an average raw error of zero, even though both predictions are wrong. Squaring solves this by making all errors positive. It also has a second effect: large errors get penalized disproportionately (a $100 error becomes 10,000; a $10 error only becomes 100), which pushes the model to prioritize eliminating big mistakes over small ones. The tradeoff is that MSE is sensitive to outliers for the same reason.

5. Two features in your dataset have a correlation of 0.97. What problem does this cause and how do you fix it?

This is multicollinearity. The two features carry nearly identical information, so the model cannot reliably separate their individual contributions. The weights for both features become unstable — small changes in the training data can produce wildly different weight values between runs, making the model hard to interpret and potentially unreliable. Fix: remove one of the two features (keep the more interpretable one), or combine them. You can detect the problem upfront with a correlation matrix or by computing the Variance Inflation Factor (VIF) — values above 5–10 signal concern.

⚡ Quick Recap — Linear Regression

What it does — Fits the best straight line through data to predict a continuous number.
The formula — ŷ = w₁x₁ + ... + w_nx_n + b. A weighted sum of features plus a baseline.
w (weight) — How much each feature shifts the prediction. One per input feature. Learned from data.
b (bias) — The baseline prediction when all inputs are zero. The y-intercept.
MSE — Loss function. Squares prediction errors, penalizing large mistakes more than small ones.
OLS — The exact closed-form solution. Computes optimal weights in one step. What sklearn uses.
R² — How much variance the model explains. 1.0 = perfect. 0.0 = no better than the mean. Negative = worse than mean.
Multicollinearity — Correlated features destabilize weights. Check correlations and remove duplicates before training.
When to use — Target is a continuous number; relationship is approximately linear.
When not to use — Curved relationships, extreme outliers, categorical targets, or highly correlated features without preprocessing.

🎯 Interview Questions

What does R² mean and what values indicate a good model?
R² measures the proportion of variance in the target explained by the model. Values near 1.0 are good; near 0.0 means the model is no better than predicting the mean; negative means it is worse than the mean. Context matters — R²=0.7 is excellent for some domains, poor for others.
What is OLS and when would you use gradient descent instead?
OLS (Ordinary Least Squares) computes the exact optimal weights in a single step via w = (XᵀX)⁻¹Xᵀy. It is fast, exact, and requires no tuning — preferred for small-to-medium datasets. Gradient descent is iterative and slower but scales better to very large datasets where matrix inversion becomes prohibitively expensive.
Why is MSE preferred over MAE in many cases?
MSE penalizes large errors more heavily due to squaring and is differentiable everywhere — which gradient descent requires. MAE is more robust to outliers and easier to interpret, but not differentiable at zero, which complicates optimization.
What are the main assumptions of linear regression?
Linearity (the relationship between features and target is linear), independence of errors, homoscedasticity (constant error variance across the range of inputs), approximate normality of residuals, and no severe multicollinearity between features.
How does regularization (Ridge / Lasso) help linear regression?
Regularization adds a penalty to the loss function that discourages large weights, reducing overfitting. Ridge (L2) shrinks all weights proportionally toward zero. Lasso (L1) can drive individual weights to exactly zero, performing automatic feature selection — useful when you suspect many features are irrelevant.

📎 Attachments & Further Reading

No attachments for this topic yet.

#LinearRegression#MLModels#OLS#GradientDescent#Regression#30DaysOfAI#Ramadan2026#sklearn