Naive Bayes:
The Model That Works Because It's Wrong

You open your email. 200 new messages. Your brain instantly sorts:
"FREE GIFT," "Claim your prize," "Nigerian prince" → spam.
"Team standup at 10am," "Invoice attached," "Happy birthday!" → not spam.

You didn't read those emails carefully. You pattern-matched on a few words.
Naive Bayes does exactly this — and it does it with math.
It makes an assumption that is almost never true, and achieves 99% accuracy anyway.

01 — What Bayes' Theorem Actually Says

Before the algorithm, understand the theorem it's built on. Bayes' theorem answers one type of question: given that I've observed some evidence, how should I update my belief about a hypothesis?

In email terms: given that this email contains the word "FREE," what is the probability it is spam?

P(spam | "FREE") = P("FREE" | spam) × P(spam) / P("FREE")

02 — The "Naive" Assumption: Multiplying Independent Probabilities

A real email has many words. Applying Bayes' theorem properly to multiple words at once would require knowing the probability of every possible combination of words appearing together — for a vocabulary of 50,000 words, that's an astronomically large table. Completely impractical.

Naive Bayes makes one bold simplification that makes the problem tractable: assume all features (words) are conditionally independent of each other given the class label. Treat each word as a completely separate, independent piece of evidence. Then just multiply all the individual probabilities together.

This is "naive" because "FREE" and "MONEY" are obviously not independent — spammers deliberately write them together. But notice: the classification was still correct. The probabilities are wrong (0.144 is not the true probability of spam) but the ranking is right — spam scored higher than not-spam, and that's all we need.

03 — The Zero Problem and Laplace Smoothing

There's a critical bug hiding in the multiplication approach. Suppose the word "CRYPTOCURRENCY" never appeared in any training spam email. Then:

P("CRYPTOCURRENCY" | spam) = 0/total_spam_words = 0.0

And now every spam score for any email containing "CRYPTOCURRENCY" becomes:

P(spam) × ... × 0.0 × ... = 0.0

One unseen word wipes out all other evidence, no matter how many other strong spam signals the email contains. This is called the zero-frequency problem.

04 — Three Types of Naive Bayes: Which to Use When

05 — From Scratch: Hand-Computing Spam Detection

Before sklearn, let's compute probabilities manually on a tiny training set. This makes every step of the algorithm visible.

from collections import defaultdict
import math

# Training data — 4 emails: 1=spam, 0=not spam
train = [
    ("free money win prize",   1),
    ("claim free gift now",    1),
    ("meeting agenda friday",  0),
    ("invoice project report", 0),
]

# Step 1: count words per class
word_counts = {0: defaultdict(int), 1: defaultdict(int)}
class_totals = {0: 0, 1: 0}
class_doc_counts = {0: 0, 1: 0}
vocabulary = set()

for text, label in train:
    words = text.split()
    class_doc_counts[label] += 1
    for word in words:
        word_counts[label][word] += 1
        class_totals[label] += 1
        vocabulary.add(word)

V = len(vocabulary)   # vocabulary size (for Laplace smoothing)
total_docs = len(train)

# Step 2: define log-probability function (uses Laplace smoothing)
# We use log-probabilities to avoid floating-point underflow from
# multiplying many tiny probabilities (0.001 × 0.002 × ... → 0.0)
def log_prob_word(word, label):
    count = word_counts[label].get(word, 0)
    return math.log((count + 1) / (class_totals[label] + V))   # Laplace

# Step 3: classify a new email
def classify(email):
    words = email.split()
    scores = {}
    for label in [0, 1]:
        # Prior: log P(class)
        score = math.log(class_doc_counts[label] / total_docs)
        # Likelihood: add log P(word | class) for each word
        for word in words:
            score += log_prob_word(word, label)
        scores[label] = score
    prediction = 1 if scores[1] > scores[0] else 0
    return prediction, scores

pred, scores = classify("free money report")
print(f"Email: 'free money report'")
print(f"  Log-score spam    = {scores[1]:.3f}")
print(f"  Log-score not spam = {scores[0]:.3f}")
print(f"  Prediction: {'Spam' if pred else 'Not spam'}")

06 — Spam Classifier in sklearn

The sklearn version handles tokenization, counting, smoothing, and log-probabilities automatically. CountVectorizer converts raw text into word-count vectors — it builds a vocabulary from all training emails, then represents each email as a list of how many times each vocabulary word appears. That representation is exactly what MultinomialNB expects.

from sklearn.naive_bayes import MultinomialNB
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.metrics import classification_report

# Training corpus — in production you'd have thousands of emails
emails = [
    "free money click now win prize",   # spam
    "free gift claim your reward now",  # spam
    "win cash prize limited offer",    # spam
    "meeting at 3pm agenda attached",   # not spam
    "invoice for project milestone",    # not spam
    "quarterly report due friday",     # not spam
]
labels = [1, 1, 1, 0, 0, 0]   # 1=spam, 0=not spam

# CountVectorizer: text → word-count vectors
# Each email becomes a row: [count("agenda"), count("attached"), ...]
vec   = CountVectorizer()
X     = vec.fit_transform(emails)
model = MultinomialNB(alpha=1.0).fit(X, labels)   # alpha=1 = Laplace smoothing

# Classify new emails
test_emails = [
    "claim your free money reward",
    "project deadline reminder friday",
]
test_X = vec.transform(test_emails)
predictions = model.predict(test_X)
probabilities = model.predict_proba(test_X)

for email, pred, proba in zip(test_emails, predictions, probabilities):
    label = "Spam" if pred else "Not spam"
    print(f"'{email}'")
    print(f"  → {label}  (spam prob: {proba[1]:.1%})")

07 — Why It Still Powers Production at Scale

08 — A Thought to Sit With

09 — Coming Up Next

Tomorrow — Day 8: Decision Trees. We move from probabilistic models to rule-based ones. Decision trees split data by asking yes/no questions — and they're the foundational building block for the most powerful tabular ML methods: Random Forests and XGBoost.

10 — Go Deeper

▶ StatQuest: Naive Bayes, Clearly Explained
YouTube · 15 min · The best walkthrough of priors, likelihoods, and posteriors — with visual examples that make Bayes' theorem click.

▶ Kaggle: Intro to Machine Learning
Free interactive course · Apply Naive Bayes and other classifiers to real datasets in your browser — the fastest bridge from reading to hands-on practice.