NLP Fundamentals:
The Complete Guide to Teaching Machines to Read

In 1957, linguist John Firth wrote:
"You shall know a word by the company it keeps."

He meant that the meaning of a word is defined by
the other words that appear alongside it in real usage.
"Bank" next to "river" means something different than "bank" next to "loan."

This one sentence contains the seed of every NLP technique
from bag-of-words to BERT — sixty-five years of progress,
all following the same original intuition.

01 — The Core Problem: Text Is Not a Number

Every ML model works on numbers. Decision trees split on numerical thresholds. Neural networks multiply weights by values. Even images, which feel qualitative, are grids of pixel intensities from 0 to 255. Text has no natural numerical representation — a word is a symbol, and symbols have no inherent magnitude, distance, or order.

The entire field of NLP is, at its core, a long answer to one question: how do you represent language as numbers in a way that captures meaning? Every technique from bag-of-words in 1954 to transformer embeddings in 2018 is a different answer to that same question. The progression goes:

Raw text
↓ tokenisation
Token list → integer IDs
↓ bag-of-words / TF-IDF
Sparse vectors (50,000-dim, mostly zeros)
↓ word2vec / GloVe — Day 23
Dense vectors (300-dim, every value non-zero)
↓ BERT / GPT — Day 26
Contextualised dense vectors (meaning changes with context)

Each step captures more meaning at the cost of more computation. Crucially, the foundational techniques are still widely used in production — not because they're "good enough," but because they're fast, interpretable, and effective for many real-world tasks. Understanding them makes the whole field click into place.

02 — Text Preprocessing: Cleaning Before You Count

Raw text is messy. Before any numerical representation can be computed, you need to normalise the text so that equivalent forms of the same word are treated identically, and noise that adds no meaning is removed. These steps collectively form the preprocessing pipeline — every classical NLP system runs through all of them in sequence.

Lowercasing

The simplest step with the largest impact. Without it, "Python," "PYTHON," and "python" are three different tokens. Lower-casing collapses them to one. The exception: acronyms (NLP vs nlp), proper nouns in some tasks, and sentiment-sensitive capitalisation ("GREAT" vs "great"). Apply lowercasing by default; reconsider only when case carries meaning for your specific task.

Punctuation and Special Character Removal

Punctuation rarely carries semantic weight for classification tasks. Removing it prevents "word." and "word" from being treated as different tokens. Exceptions: emoji (carry strong sentiment), URLs and email addresses (may be signals in spam detection), code and technical strings (remove the whole token if irrelevant, or keep as a special token). Regex-based removal is standard: re.sub(r'[^a-zA-Z0-9\s]', '', text).

Stop Word Removal

Stop words — "the," "is," "and," "a," "in," "of," "to" — appear in virtually every document and carry no discriminative information for most classification tasks. Removing them dramatically reduces vocabulary size and noise. NLTK and spaCy ship predefined stop word lists. Important caveat: do not remove stop words when word order matters (sentiment analysis with negation — "not good" becomes "good" after removing "not"). The decision is task-dependent.

Stemming vs Lemmatisation

Both normalise inflected word forms to a common base, collapsing "run," "runs," "running," "ran" into one token. They differ in how:

Rule of thumb: use stemming when speed matters and morphology is regular (English classification pipelines). Use lemmatisation when correctness matters or when the processed text will be shown to humans (keyword extraction, summarisation).

Text Normalisation

Beyond lowercasing and stop words, normalisation handles language quirks: expanding contractions ("don't" → "do not"), removing accents for multilingual text ("café" → "cafe"), handling Unicode (normalise to NFC/NFD), collapsing repeated characters ("sooooo good" → "so good"), replacing numbers with a special token (<NUM>), and stripping HTML tags from scraped web content. spaCy's pipeline handles most of these automatically for 70+ languages.

03 — Tokenisation: Splitting Text into Meaningful Units

Tokenisation is the process of splitting text into the atomic units your model will reason over. It sounds trivial but has genuine complexity: "don't" → ["don't"] or ["do", "n't"]? "New York" → one token or two? "2,500.00" → one token or three? Chinese text has no spaces at all.

04 — Bag-of-Words and N-grams: Representing Documents as Counts

Once you have tokens, the simplest numerical representation is to count them. Bag-of-Words (BoW) represents a document as a vector of length equal to the vocabulary size, where each position holds the count of the corresponding word. The name "bag" is deliberate: the order is thrown away, only the contents matter.

Example: vocabulary = ["cat", "dog", "love", "hate", "I"], document = "I love my cat and my cat loves me":

cat=2   dog=0   love=1   hate=0   I=1
→ vector: [2, 0, 1, 0, 1] (50,000-dim in practice, ~49,995 zeros)

What BoW loses: word order entirely. "The dog bit the man" and "The man bit the dog" produce identical vectors. "Not good" and "Good not" are the same. For topic classification, this rarely matters — topic signal lives in which words appear, not their order. For sentiment, reasoning, and translation, it matters enormously.

N-grams: Recovering Some Word Order

N-grams extend BoW by treating consecutive word sequences as single tokens. A bigram (n=2) treats each adjacent pair as one unit; a trigram (n=3) treats each triple. "New York" as a bigram won't be split into "New" and "York" — their co-occurrence is preserved.

In practice, ngram_range=(1,2) (unigrams + bigrams) is the standard setting. It adds phrase-level sensitivity without the vocabulary explosion of trigrams. Adding bigrams typically improves sentiment analysis accuracy by 2–5% — the improvement comes specifically from negation capture ("not good" is now a single feature with its own weight).

05 — TF-IDF: Making the Right Words Louder

Raw word counts are misleading. "The" appearing 50 times in a document is less informative than "photosynthesis" appearing 3 times. You want to weight terms not just by how often they appear in this document, but inversely by how common they are across all documents. That is exactly what TF-IDF does.

The Formula, Explained

Term Frequency (TF) measures how often term t appears in document d, normalised by document length to avoid favouring longer documents:

TF(t, d) = count(t in d) / total_tokens(d)

Inverse Document Frequency (IDF) penalises terms that appear in many documents. A term in every document carries no discriminative information:

IDF(t) = log( N / (1 + df(t)) ) + 1
N = total documents, df(t) = documents containing t
(+1 smoothing prevents division by zero for new terms)

TF-IDF(t, d) = TF(t, d) × IDF(t). A word appearing frequently in this document but rarely across the corpus scores highest — it is the document's signature term.

Worked Example

Corpus of 3 documents: "I love cats", "I love dogs", "Cats and dogs are pets". Vocabulary after stop word removal: {love, cats, dogs, pets}. N=3 documents.

Notice "pets" has the highest IDF (1.41) — it only appears in one document, making it maximally discriminative. If Doc 1 also contained "pets," it would score higher there than "love" or "cats," correctly identifying it as a distinguishing term. This is the core intuition: rare across the corpus, frequent in this document = important for this document.

06 — Beyond Counting: POS Tagging, NER, and Dependency Parsing

Bag-of-words and TF-IDF treat text as an unordered set of tokens. But language has structure — words have grammatical roles, entities have types, clauses have relationships. The classical NLP pipeline uses three additional annotation layers to extract this structure:

Part-of-Speech (POS) Tagging

Assigns a grammatical category to each token: noun (NN), verb (VB), adjective (JJ), adverb (RB), preposition (IN), etc. "The quick brown fox jumps." → [("The", DT), ("quick", JJ), ("brown", JJ), ("fox", NN), ("jumps", VBZ)]. Why it matters: (1) lemmatisation requires POS — "better" as adjective lemmatises to "good," as verb to "better"; (2) feature engineering — count only noun phrases for entity-heavy tasks; (3) disambiguating word sense — "book a flight" (VB) vs "read a book" (NN). spaCy's POS tagger achieves ~97% accuracy on standard English benchmarks using a CNN-based model.

Named Entity Recognition (NER)

Identifies and classifies real-world entities: persons (PER), organisations (ORG), locations (LOC), dates (DATE), monetary values (MONEY), etc. "Apple Inc. announced a $3B deal in Cupertino yesterday." → [(Apple Inc., ORG), ($3B, MONEY), (Cupertino, LOC), (yesterday, DATE)]. NER powers: information extraction from news and reports, knowledge graph construction, question answering ("Where is Tesla headquartered?"), and compliance systems that must detect mentions of specific entities.

Dependency Parsing

Identifies grammatical relationships between words: which noun is the subject of which verb? Which adjective modifies which noun? "The angry customer called the support team." → "customer" is the subject (nsubj) of "called"; "angry" modifies "customer" (amod); "team" is the object (dobj) of "called." Dependency parsing enables: semantic role labelling, relation extraction ("Company X acquired Company Y"), coreference resolution ("Obama announced his plan" — "his" refers to "Obama"), and question answering from structured text.

07 — Classical Text Classification: The Full Pipeline

Putting all the preprocessing steps together with a classifier produces a complete, deployable text classification system. This is what a production pipeline looks like — the same architecture Gmail used for spam filtering, the same one most enterprise NLP systems use for routing and categorisation today.

08 — Code: Complete NLP Pipeline with spaCy and scikit-learn

The code below implements the full pipeline — spaCy-based preprocessing including lemmatisation and stop word removal, TF-IDF feature extraction with bigrams, three classifiers compared, and feature inspection. This is a production-quality template.

import re, spacy
from sklearn.feature_extraction.text import TfidfVectorizer, CountVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.naive_bayes import MultinomialNB
from sklearn.svm import LinearSVC
from sklearn.pipeline import Pipeline
from sklearn.datasets import fetch_20newsgroups
from sklearn.metrics import classification_report, accuracy_score
import numpy as np

# ── Load spaCy model ─────────────────────────────────────────────────────
# python -m spacy download en_core_web_sm
nlp = spacy.load('en_core_web_sm', disable=['parser', 'ner'])  # fast: only tagger

def preprocess(text):
    """Full preprocessing: lowercase, clean, lemmatise, remove stops."""
    text = text.lower()
    text = re.sub(r'http\S+|www\S+', ' ', text)        # strip URLs
    text = re.sub(r'[^a-z0-9\s]', ' ', text)           # remove punctuation
    text = re.sub(r'\s+', ' ', text).strip()           # collapse whitespace
    doc = nlp(text)
    tokens = [
        token.lemma_ for token in doc
        if not token.is_stop
        and not token.is_punct
        and token.is_alpha
        and len(token.text) > 2
    ]
    return ' '.join(tokens)

# ── Load 20 Newsgroups (4 categories) ───────────────────────────────────
cats = ['sci.space', 'talk.religion.misc', 'comp.graphics', 'rec.sport.hockey']
train = fetch_20newsgroups(subset='train', categories=cats, remove=('headers','footers'))
test  = fetch_20newsgroups(subset='test',  categories=cats, remove=('headers','footers'))

print(f"Train: {len(train.data)} docs | Test: {len(test.data)} docs")

# Preprocess (takes ~30s with spaCy lemmatisation)
train_clean = [preprocess(t) for t in train.data]
test_clean  = [preprocess(t) for t in test.data]

# ── TF-IDF vectoriser ────────────────────────────────────────────────────
tfidf = TfidfVectorizer(
    ngram_range   = (1, 2),      # unigrams + bigrams
    max_features  = 50000,       # cap vocabulary size
    min_df        = 2,           # ignore tokens in < 2 docs (noise)
    sublinear_tf  = True,        # log(1 + tf) — reduce impact of high counts
    strip_accents = 'unicode',   # normalise accented chars
)
X_train = tfidf.fit_transform(train_clean)   # fit only on train
X_test  = tfidf.transform(test_clean)        # never fit_transform on test
print(f"Vocabulary size: {len(tfidf.vocabulary_):,}")
print(f"Matrix shape: {X_train.shape}  (sparse: {X_train.nnz / X_train.shape[0] / X_train.shape[1]:.4%} non-zero)")

# ── Compare three classifiers ────────────────────────────────────────────
models = {
    'Logistic Regression': LogisticRegression(C=5.0, max_iter=1000, solver='saga'),
    'Naive Bayes':         MultinomialNB(alpha=0.1),
    'Linear SVM':          LinearSVC(C=1.0, max_iter=2000),
}
for name, clf in models.items():
    clf.fit(X_train, train.target)
    preds = clf.predict(X_test)
    acc = accuracy_score(test.target, preds)
    print(f"{name:22s}  accuracy={acc:.4f}")

# ── Full report for best model (SVM) ────────────────────────────────────
best = models['Linear SVM']
preds = best.predict(X_test)
print("\n" + classification_report(test.target, preds, target_names=cats))

# ── Inspect top features per class ───────────────────────────────────────
feat_names = np.array(tfidf.get_feature_names_out())
for i, cat in enumerate(cats):
    top_idx = best.coef_[i].argsort()[-8:][::-1]
    print(f"{cat}: {', '.join(feat_names[top_idx])}")

# ── NER demo with spaCy ───────────────────────────────────────────────────
nlp_full = spacy.load('en_core_web_sm')   # full pipeline with NER
sample = "NASA announced a $2.5B contract with SpaceX for the Artemis mission in Houston."
doc = nlp_full(sample)
for ent in doc.ents:
    print(f"  {ent.text:20s} → {ent.label_} ({spacy.explain(ent.label_)})")

09 — When to Use What: Comparing All Representations

Classical NLP offers several text representation strategies. The right choice depends on your task, data size, latency requirements, and interpretability needs. Here is the complete comparison:

10 — The Fundamental Limitations — and Why They Lead to Embeddings

Understanding what classical NLP cannot do is as important as understanding what it can. These limitations are not engineering failures — they are principled consequences of the bag-of-words assumption, and each one motivated a specific next technique.

11 — Why Classical NLP Still Powers Production Systems

The year is 2026. Transformers dominate NLP benchmarks. Yet classical NLP techniques run at the core of many of the largest systems in production. Understanding why tells you something important about engineering.

BM25 runs the first retrieval stage in most RAG pipelines. Every "ChatGPT with web search" system uses BM25 (or an equivalent sparse retrieval method) to do a fast first-pass retrieval over millions of documents in milliseconds. Neural retrieval (dense embeddings) is far too slow to run first at scale. BM25 finds the candidates; embeddings rerank the shortlist.

TF-IDF classifiers power industrial log analysis. A server generating 10 million log lines per day cannot run BERT on each line — latency and cost are prohibitive. TF-IDF + Naive Bayes classifies log severity in microseconds, routing critical alerts in real time. For this task it is not a "weaker backup" — it is the correct engineering choice.

Sparse representations are more interpretable in regulated industries. Healthcare, legal, and financial compliance systems must explain predictions. "This contract was flagged because it contains the terms 'termination,' 'liability waiver,' and 'governing law' with high TF-IDF scores" satisfies a compliance audit. "The transformer's attention weights suggest..." does not.

Classical preprocessing is still required for transformers. BERT doesn't take raw text — it takes subword tokens produced by WordPiece tokenisation, a direct descendant of the techniques in this post. The preprocessing pipeline (cleaning, normalisation, tokenisation) is still step one for every modern NLP system. You can't skip it just because the model is more powerful.

12 — A Thought to Sit With

13 — Coming Up Next

Tomorrow — Day 20: Recurrent Neural Networks. Classical NLP treats language as bags of tokens — order doesn't matter, memory doesn't exist. The RNN was the first serious attempt to fix that: a network that reads a sequence word by word, carrying a hidden state forward as it goes, so that every word is understood in light of everything that came before it. We'll build the RNN cell from scratch, trace the forward pass, derive exactly why gradients vanish over long sequences — and understand why that one mathematical problem drove 20 years of research leading straight to the transformer.

14 — Go Deeper

▶ TF-IDF Explained Visually — Computerphile
YouTube · 12 min · Clear walkthrough of the formula with worked examples and search engine context.

🔗 spaCy Linguistic Features — Official Docs
Complete reference for POS tagging, NER, dependency parsing, and lemmatisation in spaCy. The best NLP library for production pipelines.

🔗 sklearn Text Feature Extraction — Official Docs
Full API reference for TfidfVectorizer, CountVectorizer, HashingVectorizer with examples.

📄 TF-IDF and BM25 Lecture Notes — Cambridge University
PDF · The mathematical derivation of BM25 from first principles. Section 3 explains the saturation and length-normalisation motivation clearly.

📄 Efficient Estimation of Word Representations — Mikolov et al. (2013)
The Word2Vec paper — a direct preview of Day 23. Understanding the motivation section (why BoW falls short) is essential context for embeddings.