Neural Networks:
From Logistic Regression to Deep Learning

In 1943, Warren McCulloch and Walter Pitts described a mathematical model
of how a brain cell fires: if the sum of its inputs exceeds a threshold, it activates.
Otherwise it stays silent.

They had no idea they were describing
what would become the architecture behind GPT-4.

The idea is that simple. The consequences are that large.

01 — The Bridge: You Already Know Most of This

Before writing a single line of neural network code, let's locate where you actually are. Over the last 14 days you've learned: how gradient descent finds optimal weights (Day 3), how logistic regression maps a weighted sum of inputs to a probability (Day 5), how regularisation prevents overfitting (Day 10), and how ensembles of weak models build something stronger (Days 11–13).

A neural network uses all of these ideas simultaneously. It is, at its core, nothing more than a logistic regression unit — extended in two ways. First, you apply many of them in parallel, each looking at the input from a different angle. Second, you feed those intermediate outputs into another set of logistic regression units — creating a hierarchy of learned representations. That hierarchy is what makes neural networks universally powerful. The individual components are things you've already seen.

02 — The Biological Neuron vs the Mathematical Neuron

The biological inspiration is worth understanding — not because the analogy is perfect, but because it explains why the architecture is shaped the way it is.

The key insight: the biological and mathematical versions are structurally identical. Dendrites = input weights. Cell body summation = weighted sum z. Firing threshold = activation function. Axon output = activation a = f(z). The translation from biology to algebra is almost one-to-one.

03 — The Neuron Equation: From Logistic Regression to General Neuron

This is the central connection of the entire post. Look at these two equations side by side:

04 — Why One Neuron Is Not Enough: The XOR Problem

A single neuron — regardless of what activation function it uses — can only learn a linear decision boundary. It draws one straight line (or hyperplane in higher dimensions) and classifies everything on one side as one class, and the other side as another class. This is the exact same limitation as logistic regression.

The XOR problem (exclusive OR) is the classic demonstration of why this is limiting:

import numpy as np
import matplotlib
matplotlib.use('Agg')
from sklearn.linear_model import LogisticRegression
from sklearn.neural_network import MLPClassifier

# XOR: output is 1 when exactly one input is 1 — NOT linearly separable
X_xor = np.array([[0,0], [0,1], [1,0], [1,1]])
y_xor = np.array([0,   1,   1,   0  ])

# Single logistic regression unit (one neuron, sigmoid)
lr = LogisticRegression()
lr.fit(X_xor, y_xor)

# Neural network: 2 hidden neurons → ReLU → 1 output neuron
nn = MLPClassifier(hidden_layer_sizes=(2,), activation='relu',
                    max_iter=10000, random_state=42)
nn.fit(X_xor, y_xor)

print(f"{'Input':>10}  {'True':>5}  {'Logistic Reg':>14}  {'Neural Net':>11}")
print("-" * 50)
for x, y in zip(X_xor, y_xor):
    lr_pred = lr.predict([x])[0]
    nn_pred = nn.predict([x])[0]
    print(f"{str(x):>10}  {y:>5}  {lr_pred:>14}  {nn_pred:>11}")

print(f"\nLogistic Regression accuracy: {lr.score(X_xor, y_xor):.0%}")
print(f"Neural Network accuracy:      {nn.score(X_xor, y_xor):.0%}")

05 — Architecture: Layers, Weights, and Shapes

A neural network is a directed graph of neurons arranged in layers. Data flows left to right — from input layer through hidden layers to output layer. Every arrow between neurons represents a weight.

import numpy as np

# Build a 2-layer neural network from scratch
# Architecture: 2 inputs → 3 hidden neurons → 1 output

np.random.seed(42)

# ── Weight matrices (randomly initialised) ──────────────────────────────
W1 = np.random.randn(2, 3) * 0.1   # shape: (n_inputs=2, n_hidden=3)
b1 = np.zeros(3)                   # shape: (n_hidden=3,)
W2 = np.random.randn(3, 1) * 0.1   # shape: (n_hidden=3, n_output=1)
b2 = np.zeros(1)                   # shape: (n_output=1,)

# ── Activation functions ─────────────────────────────────────────────────
def sigmoid(z): return 1 / (1 + np.exp(-z))
def relu(z):    return np.maximum(0, z)

# ── Forward pass ─────────────────────────────────────────────────────────
def forward(X, W1, b1, W2, b2):
    Z1 = X @ W1 + b1            # (batch, 2) @ (2, 3) → (batch, 3)
    A1 = relu(Z1)               # non-linearity in hidden layer
    Z2 = A1 @ W2 + b2           # (batch, 3) @ (3, 1) → (batch, 1)
    A2 = sigmoid(Z2)            # sigmoid at output → probability [0,1]
    return Z1, A1, Z2, A2

# ── Single example ───────────────────────────────────────────────────────
X_sample = np.array([[0.5, -1.2]])   # 1 example, 2 features
Z1, A1, Z2, A2 = forward(X_sample, W1, b1, W2, b2)

print("── Forward Pass (single example) ──")
print(f"Input X:          {X_sample[0]}")
print(f"Z1 (pre-relu):    {Z1[0].round(4)}")
print(f"A1 (after relu):  {A1[0].round(4)}  ← negative values zeroed")
print(f"Z2 (pre-sigmoid): {Z2[0].round(4)}")
print(f"A2 (output prob): {A2[0].round(4)}  ← final prediction")
print(f"\nShapes: X={X_sample.shape}, W1={W1.shape}, A1={A1.shape}, W2={W2.shape}, A2={A2.shape}")

# Count parameters
n_params = W1.size + b1.size + W2.size + b2.size
print(f"Total learnable parameters: {n_params}  (W1:{W1.size} + b1:{b1.size} + W2:{W2.size} + b2:{b2.size})")

06 — Activation Functions: The Non-Linearity Menu

The activation function is what makes neural networks powerful. Without it — if every layer just applied a linear transformation — you could collapse the entire network into a single linear layer, no matter how many layers you stacked. Activation functions break this collapse by introducing non-linearity at every neuron.

import numpy as np

x = np.linspace(-3, 3, 7)  # 7 sample values from -3 to 3

# ── All common activation functions ──────────────────────────────────────
sigmoid  = lambda z: 1 / (1 + np.exp(-z))
tanh     = lambda z: np.tanh(z)
relu     = lambda z: np.maximum(0, z)
leaky    = lambda z: np.where(z > 0, z, 0.01 * z)
gelu     = lambda z: z * sigmoid(1.702 * z)   # approximate GELU
softmax  = lambda z: np.exp(z) / np.exp(z).sum() # for output layer, multi-class

print(f"{'x':>6}  {'sigmoid':>8}  {'tanh':>7}  {'relu':>6}  {'leaky':>7}  {'gelu':>7}")
print("-" * 55)
for xi in x:
    print(f"{xi:>6.1f}  {sigmoid(xi):>8.4f}  {tanh(xi):>7.4f}  {relu(xi):>6.1f}  {leaky(xi):>7.4f}  {gelu(xi):>7.4f}")

print(f"\nKey derivatives at x=0 (gradient flow):")
print(f"  sigmoid'(0) = 0.25   ← non-zero but small, vanishes at extremes")
print(f"  tanh'(0)    = 1.0    ← stronger gradient at centre")
print(f"  relu'(0)    = 0 or 1 ← 0 for negatives (dead neurons), 1 for positives")
print(f"  gelu'(0)    ≈ 0.5    ← smooth transition, used in Transformers/GPT")

07 — Backpropagation: How Gradients Flow Backwards

Training a neural network means adjusting all the weights to minimise the loss. Gradient descent needs the gradient of the loss with respect to every weight. Backpropagation is the efficient algorithm for computing all these gradients simultaneously using the chain rule of calculus.

import numpy as np

np.random.seed(42)

# ── Dataset: XOR problem ─────────────────────────────────────────────────
X = np.array([[0,0], [0,1], [1,0], [1,1]], dtype=float)
y = np.array([[0], [1], [1], [0]], dtype=float)

# ── Weight initialisation ─────────────────────────────────────────────────
W1 = np.random.randn(2, 4) * 0.5  # 2 inputs → 4 hidden
b1 = np.zeros((1, 4))
W2 = np.random.randn(4, 1) * 0.5  # 4 hidden → 1 output
b2 = np.zeros((1, 1))

sigmoid  = lambda z: 1 / (1 + np.exp(-z))
relu     = lambda z: np.maximum(0, z)
relu_d   = lambda z: (z > 0).astype(float)     # derivative of ReLU
bce_loss = lambda y, yh: -np.mean(y*np.log(yh+1e-8) + (1-y)*np.log(1-yh+1e-8))

lr = 0.5

print(f"{'Epoch':>6} {'Loss':>8} {'Predictions':>30}")
print("-" * 52)

for epoch in range(5001):
    # ── Forward pass ────────────────────────────────────────────────────
    Z1 = X @ W1 + b1
    A1 = relu(Z1)
    Z2 = A1 @ W2 + b2
    A2 = sigmoid(Z2)

    loss = bce_loss(y, A2)

    # ── Backward pass (chain rule) ──────────────────────────────────────
    dA2 = A2 - y                          # ∂Loss/∂A2 (BCE + sigmoid combined)
    dW2 = A1.T @ dA2 / len(X)            # ∂Loss/∂W2
    db2 = dA2.mean(axis=0, keepdims=True)

    dA1 = dA2 @ W2.T                      # propagate error back
    dZ1 = dA1 * relu_d(Z1)               # gate through ReLU derivative
    dW1 = X.T @ dZ1 / len(X)             # ∂Loss/∂W1
    db1 = dZ1.mean(axis=0, keepdims=True)

    # ── Weight update ───────────────────────────────────────────────────
    W2 -= lr * dW2;  b2 -= lr * db2
    W1 -= lr * dW1;  b1 -= lr * db1

    if epoch % 1000 == 0:
        preds = (A2 > 0.5).astype(int).flatten()
        print(f"{epoch:>6} {loss:>8.4f}   {preds.tolist()} vs {y.flatten().astype(int).tolist()}")

print(f"\nFinal predictions (probabilities):")
for xi, yi, ai in zip(X, y, A2):
    print(f"  Input {xi.astype(int)} → prob={ai[0]:.4f} → pred={int(ai[0]>0.5)} (true={int(yi[0])})")

08 — The Vanishing Gradient Problem

Now that you've seen backpropagation, you can understand the most important historical obstacle in deep learning — and why ReLU solved it.

import numpy as np

# Demonstrate how gradients shrink through sigmoid layers
# At each layer, sigmoid derivative is max 0.25
sigmoid_d = lambda z: sigmoid(z) * (1 - sigmoid(z))  # max 0.25
sigmoid   = lambda z: 1 / (1 + np.exp(-z))
relu_d    = lambda z: (z > 0).astype(float)

print("Gradient magnitude after backprop through N layers")
print(f"{'Layers':>7}  {'Sigmoid gradient':>18}  {'ReLU gradient':>15}")
print("-" * 48)

grad = 1.0
for L in range(1, 11):
    sig_grad  = 0.25 ** L       # worst case: max derivative per layer
    relu_grad = 1.0 ** L         # for active neurons: derivative = 1
    print(f"{L:>7}  {sig_grad:>18.8f}  {relu_grad:>15.4f}")

print(f"\nSigmoid gradient after 10 layers: {0.25**10:.2e} (essentially zero)")
print(f"ReLU gradient after 10 layers:    {1.0**10:.4f} (unchanged for active neurons)")

09 — Complete Neural Network with PyTorch

Now the full picture — from data to trained model to evaluation, using PyTorch (the dominant deep learning framework):

import torch
import torch.nn as nn
import torch.optim as optim
import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import accuracy_score, classification_report

# ── Data preparation ─────────────────────────────────────────────────────
data = load_breast_cancer()
X, y = data.data, data.target

scaler = StandardScaler()                           # Neural networks need scaled features
X = scaler.fit_transform(X)

X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, random_state=42)

X_tr = torch.FloatTensor(X_tr)
X_te = torch.FloatTensor(X_te)
y_tr = torch.FloatTensor(y_tr).unsqueeze(1)        # shape: (N, 1)
y_te = torch.FloatTensor(y_te).unsqueeze(1)

# ── Model definition ─────────────────────────────────────────────────────
class NeuralNet(nn.Module):
    def __init__(self, n_features):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(n_features, 64),     # input → hidden 1 (64 neurons)
            nn.ReLU(),
            nn.Dropout(0.2),              # randomly zero 20% of neurons during training
            nn.Linear(64, 32),            # hidden 1 → hidden 2 (32 neurons)
            nn.ReLU(),
            nn.Dropout(0.2),
            nn.Linear(32, 1),             # hidden 2 → output (1 neuron)
            nn.Sigmoid()                   # output probability [0,1]
        )

    def forward(self, x):
        return self.net(x)

model     = NeuralNet(X_tr.shape[1])           # 30 input features
criterion = nn.BCELoss()                         # binary cross-entropy
optimizer = optim.Adam(model.parameters(), lr=0.001)

# ── Count parameters ─────────────────────────────────────────────────────
n_params = sum(p.numel() for p in model.parameters())
print(f"Model: {n_params} trainable parameters")
print(f"Architecture: 30 → 64 → 32 → 1  (ReLU + Dropout + Sigmoid)\n")

# ── Training loop ─────────────────────────────────────────────────────────
print(f"{'Epoch':>6}  {'Train Loss':>11}  {'Train Acc':>10}  {'Val Acc':>8}")
print("-" * 45)

for epoch in range(1, 101):
    model.train()
    optimizer.zero_grad()
    y_pred = model(X_tr)
    loss   = criterion(y_pred, y_tr)
    loss.backward()                              # compute all gradients (backprop)
    optimizer.step()                             # update all weights

    if epoch % 20 == 0:
        model.eval()
        with torch.no_grad():
            tr_acc = ((y_pred > 0.5) == y_tr).float().mean().item()
            val_acc = ((model(X_te) > 0.5) == y_te).float().mean().item()
        print(f"{epoch:>6}  {loss.item():>11.4f}  {tr_acc:>10.4f}  {val_acc:>8.4f}")

# ── Final evaluation ─────────────────────────────────────────────────────
model.eval()
with torch.no_grad():
    y_pred_te = (model(X_te) > 0.5).numpy().astype(int).flatten()

print(f"\nFinal test accuracy: {accuracy_score(y_te.numpy(), y_pred_te):.4f}")
print("\nClassification Report:")
print(classification_report(y_te.numpy().astype(int), y_pred_te,
      target_names=data.target_names))

10 — Neural Network vs Logistic Regression: The Full Comparison

You now have everything needed to make the comparison precise:

11 — What Comes Next: The Deep Learning Stack

Today you built the foundation. Every deep learning architecture you'll encounter in the coming days is a specialisation or extension of this same structure — neurons, layers, activations, backprop, gradient descent. Here's how what's coming connects back to today:

12 — A Thought to Sit With

13 — Coming Up Next

Day 16 — CNNs from Scratch. Convolutional Neural Networks are what made computer vision tractable. Now that you understand layers, activations, and backprop, you'll see exactly why convolutions are just a special kind of linear layer — and how pooling, receptive fields, and feature maps emerge naturally from the same principles you learned today.

14 — Go Deeper

▶ 3Blue1Brown: But what is a neural network? (Chapter 1)
YouTube · 19 min · The most beautiful visual introduction to neural network structure and forward pass. Watch before reading §05.

▶ 3Blue1Brown: Gradient descent, how neural networks learn (Chapter 2)
YouTube · 21 min · Visualises the loss landscape and gradient descent as navigation. Perfect companion to §07.

▶ 3Blue1Brown: What is backpropagation really doing? (Chapter 3)
YouTube · 14 min · The clearest visual explanation of backprop and the chain rule in existence. Watch alongside §07–08.

🔗 TensorFlow Playground — playground.tensorflow.org
Interactive browser demo: draw your own datasets, choose architectures, watch the network learn a decision boundary in real time. The best tool for building intuition about layers and activations.