Word Embeddings:
Word2Vec, GloVe & Sentence Transformers

You shall know a word
by the company it keeps.

— J.R. Firth, 1957

This single sentence, written sixty-six years before ChatGPT,
is the theoretical foundation of every embedding in existence.

01 — The Problem Token IDs Don't Solve

Yesterday we learned that tokenization converts text into numbers. Your message becomes a list of integers before the model ever processes it. "cat" → 9246. "dog" → 10077. "kitten" → 19772.

But here is the problem: those numbers are arbitrary. 9246 and 10077 are just entries in a lookup table — they carry no relationship to each other. The number 9246 is no closer to 10077 than it is to 45,000. Yet "cat" and "dog" are very closely related — both are household pets, both are animals, both appear in similar sentences. "cat" and "kitten" are even more closely related: one is literally the younger form of the other.

Raw token IDs are useless for capturing this. They are the equivalent of naming things with random serial numbers. So we need a different kind of representation — one where similar words produce similar numbers. Not a single number per word, but a list of hundreds of numbers — a vector — arranged so that words with similar meanings cluster together in space.

The vector for "cat" and the vector for "dog" are nearly parallel — they point in almost the same direction in 300-dimensional space. "Politics" points in a completely different direction. Now similarity is measurable, and arithmetic on concepts becomes possible.

02 — The Distributional Hypothesis: Meaning From Context

Before the algorithms, we need the theoretical foundation. Why should training on text co-occurrence produce vectors that capture meaning? The answer comes from linguistics, not computer science.

In 1957, linguist J.R. Firth wrote: "You shall know a word by the company it keeps." This is the distributional hypothesis — the idea that words with similar meanings tend to appear in similar contexts.

This is the magic of large-scale text. You don't need a linguist to manually write that "cat" and "kitten" are related — you just need to observe that they appear in the same contexts millions of times. The mathematical structure of co-occurrence is the structure of meaning, at least to a very good first approximation.

03 — Why One-Hot Encoding Fails

Before Word2Vec, the dominant approach was one-hot encoding. Understanding why it fails makes the embedding solution much clearer.

Every word is equally far from every other word. "cat" is as different from "dog" as it is from "galaxy." The representation carries zero semantic information.

The solution is to learn dense, low-dimensional representations: vectors of 100–1000 numbers (not 50,000), where similar words have similar vectors. This is exactly what Word2Vec introduced in 2013.

04 — Word2Vec: Learning Meaning From Prediction

In 2013, Tomas Mikolov and colleagues at Google published Word2Vec — a shockingly simple neural network that produced astonishingly good word representations. The central insight was counterintuitive: don't try to count co-occurrences. Train a network to predict words from their context, and read off the learned weights as the embeddings.

Word2Vec comes in two symmetric flavours:

The Architecture

The network is almost embarrassingly simple: input → one hidden layer (no activation) → output softmax. What matters is the task, not the architecture.

By training to predict context words from a centre word, the hidden layer weights are forced to encode what contexts each word appears in. Two words that consistently appear in similar contexts will converge toward similar weight vectors — because similar vectors produce similar predictions, and similar predictions incur similar losses on similar training examples.

The Training Trick: Negative Sampling

Computing softmax over a 50,000-word vocabulary is expensive — you normalise over all 50,000 outputs for every training step. Word2Vec uses negative sampling to avoid this: instead of predicting the full distribution, train a binary classifier. For each (centre, context) pair, ask "is this a real pair or a randomly sampled fake?"

The Famous Analogy Arithmetic

Once trained on a large corpus, something remarkable emerges. You can do arithmetic on meaning:

What Word2Vec Gets Wrong

05 — GloVe: Global Vectors for Word Representation

Word2Vec is trained locally — it looks at one (centre, context) pair at a time. Stanford researchers Pennington, Socher, and Manning argued in 2014 that this throws away global statistical information available in the full corpus. Their approach: start from the full co-occurrence matrix and factorize it directly.

The Co-Occurrence Matrix

Before training, GloVe counts how often every word pair appears within a fixed window across the entire corpus, building a matrix X where X[i][j] = times word j appears in the context of word i.

The GloVe Objective

GloVe vs Word2Vec

In practice, both produce embeddings of comparable quality for most downstream tasks. Many practitioners simply try both and pick whichever works better for their specific use case. The shared limitation — one vector per word regardless of context — is what both leave for the next generation to solve.

06 — The Polysemy Problem and Contextual Embeddings

Both Word2Vec and GloVe assign one fixed vector to each word type, regardless of context. But many words have completely different meanings in different sentences.

The solution is context-dependent embeddings: instead of one fixed vector per word type, compute a different vector for each word occurrence based on the surrounding sentence. This is what contextual embeddings do — and it's the bridge from Word2Vec/GloVe to the modern transformer era.

ELMo: The First Contextual Embeddings (2018)

ELMo (Embeddings from Language Models, AllenNLP 2018) was the first widely adopted contextual approach. It runs each sentence through a bidirectional LSTM and uses the resulting hidden states as word representations. The same word "bank" in two different sentences produces two different ELMo vectors — because the hidden states encode the sentence history.

ELMo was a major advance, but was quickly superseded by BERT (2018), which replaces LSTMs with transformers — attending to the full sentence simultaneously at every layer rather than processing left-to-right and right-to-left in two separate passes. Day 26 covers BERT and GPT in depth.

07 — Sentence Transformers: Embedding Whole Sentences

Word-level embeddings answer "what does this word mean here?" But many real applications need a different answer: "how similar is the meaning of sentence A to sentence B?" — and they need it to be fast enough to search millions of documents.

This is the problem sentence transformers solve, and it's the core technology behind semantic search, RAG retrieval, duplicate detection, and recommendation systems.

The Naive Approach and Why It Fails

Why Raw BERT Doesn't Scale to Semantic Search

Sentence-BERT: The Siamese Solution (2019)

Nils Reimers and Iryna Gurevych (UKP Lab) published Sentence-BERT (SBERT) in 2019. The solution: fine-tune BERT with a siamese network architecture using sentence pair similarity tasks, so the resulting embeddings are directly comparable by cosine similarity — without joint encoding.

How This Powers RAG (Day 29 Preview)

08 — Cosine Similarity: Measuring Meaning With Angles

Throughout this post we've talked about "similar vectors" without precisely defining similarity. The standard measure in embedding space is cosine similarity.

Why cosine and not Euclidean distance? Because cosine is magnitude-invariant. A 500-word document will have a larger embedding magnitude than a 5-word query (more vectors were pooled together), but their cosine similarity still correctly captures semantic closeness. Euclidean distance would be dominated by the length difference, not the meaning difference.

09 — Code: From Token IDs to Semantic Search

# ─────────────────────────────────────────────────────────────────────
# PART 1: The problem with raw token IDs
# ─────────────────────────────────────────────────────────────────────
import numpy as np

vocab = {"cat": 0, "dog": 1, "kitten": 2, "politics": 3, "democracy": 4}
V = len(vocab)

def one_hot(word_id, vocab_size):
    v = np.zeros(vocab_size)
    v[word_id] = 1.0
    return v

def cosine(u, v):
    return np.dot(u, v) / (np.linalg.norm(u) * np.linalg.norm(v) + 1e-10)

cat_oh    = one_hot(vocab["cat"],    V)
dog_oh    = one_hot(vocab["dog"],    V)
kitten_oh = one_hot(vocab["kitten"], V)

print(f"cos(cat, dog)    = {cosine(cat_oh, dog_oh):.3f}")    # 0.000
print(f"cos(cat, kitten) = {cosine(cat_oh, kitten_oh):.3f}") # 0.000
# All pairs are equally "different" — one-hot encodes nothing about meaning

# ─────────────────────────────────────────────────────────────────────
# PART 2: Loading GloVe embeddings and word analogy arithmetic
# ─────────────────────────────────────────────────────────────────────
# Download: https://nlp.stanford.edu/projects/glove/ (glove.6B.300d.txt)

def load_glove(path, max_words=50_000):
    """Load GloVe vectors into a dict {word: np.array}."""
    embeddings = {}
    with open(path, encoding="utf-8") as f:
        for i, line in enumerate(f):
            if i >= max_words:
                break
            values = line.split()
            word   = values[0]
            vector = np.array(values[1:], dtype=np.float32)
            embeddings[word] = vector
    return embeddings

def analogy(glove, a, b, c, topk=5):
    """Find word d such that a:b :: c:d   (d ≈ b - a + c)."""
    target  = glove[b] - glove[a] + glove[c]
    target  = target / (np.linalg.norm(target) + 1e-10)
    exclude = {a, b, c}
    scores  = [
        (word, np.dot(target, vec / (np.linalg.norm(vec) + 1e-10)))
        for word, vec in glove.items()
        if word not in exclude
    ]
    return sorted(scores, key=lambda x: -x[1])[:topk]

# glove = load_glove("glove.6B.300d.txt")
# print(analogy(glove, "man",    "king",   "woman"))  # → queen  ✓
# print(analogy(glove, "france", "paris",  "italy"))  # → rome   ✓
# print(analogy(glove, "good",   "better", "bad"))    # → worse  ✓

def nearest_neighbours(glove, word, topk=5):
    """Find the k most similar words to a query word."""
    target = glove[word] / (np.linalg.norm(glove[word]) + 1e-10)
    scores = [
        (w, np.dot(target, v / (np.linalg.norm(v) + 1e-10)))
        for w, v in glove.items()
        if w != word
    ]
    return sorted(scores, key=lambda x: -x[1])[:topk]

# nearest_neighbours(glove, "cat")
# → [("cats", 0.924), ("dog", 0.921), ("kitten", 0.908), ("feline", 0.881), ...]

# ─────────────────────────────────────────────────────────────────────
# PART 3: Sentence transformers for semantic search
# ─────────────────────────────────────────────────────────────────────
# pip install sentence-transformers

from sentence_transformers import SentenceTransformer
import numpy as np

model = SentenceTransformer("all-MiniLM-L6-v2")  # 80 MB, 384-dim, fast

# Knowledge base
documents = [
    "The Eiffel Tower is located in Paris, France.",
    "Photosynthesis converts sunlight into chemical energy in plants.",
    "Python is a high-level programming language known for readability.",
    "The mitochondria is the powerhouse of the cell.",
    "Gradient descent is an optimisation algorithm used in machine learning.",
    "The Great Wall of China stretches over 21,000 kilometres.",
    "Transformers use self-attention to process sequences in parallel.",
]

# Offline indexing (done once)
doc_embeddings = model.encode(documents, normalize_embeddings=True)
# Shape: (7, 384) — one unit vector per document

# Runtime query
query = "What optimisation algorithm is used in deep learning?"
q_emb = model.encode([query], normalize_embeddings=True)[0]

# Cosine search (dot product of unit vectors = cosine similarity)
scores = doc_embeddings @ q_emb
ranked = np.argsort(scores)[::-1]

print(f"\nQuery: {query}\n")
for rank, idx in enumerate(ranked[:3]):
    print(f"  {rank+1}. [score={scores[idx]:.3f}]  {documents[idx]}")

# Output:
# 1. [score=0.712]  Gradient descent is an optimisation algorithm used in ML.
# 2. [score=0.534]  Transformers use self-attention to process sequences.
# 3. [score=0.298]  Python is a high-level programming language...

# ─────────────────────────────────────────────────────────────────────
# PART 4: FAISS for production-scale vector search
# ─────────────────────────────────────────────────────────────────────
# pip install faiss-cpu
import faiss

dim = 384  # matches all-MiniLM-L6-v2

# Flat index: exact inner product search (cosine on unit vectors)
index = faiss.IndexFlatIP(dim)
index.add(doc_embeddings.astype("float32"))
print(f"Index size: {index.ntotal} vectors")

# Search — returns top-3
D, I = index.search(q_emb.reshape(1, -1).astype("float32"), k=3)

print("\nFAISS results:")
for score, doc_idx in zip(D[0], I[0]):
    print(f"  [score={score:.3f}]  {documents[doc_idx]}")

# For millions of documents: use IndexIVFFlat or IndexHNSWFlat
# for approximate nearest-neighbour search with sub-millisecond latency

# ─────────────────────────────────────────────────────────────────────
# PART 5: Visualising embedding clusters with t-SNE
# ─────────────────────────────────────────────────────────────────────
# pip install scikit-learn matplotlib

from sklearn.manifold import TSNE
import matplotlib.pyplot as plt

words = [
    "cat", "dog", "kitten", "puppy", "feline", "canine",      # animals (green)
    "king", "queen", "prince", "princess", "royal", "throne",  # royalty (yellow)
    "python", "java", "code", "program", "function", "debug",  # programming (blue)
]

word_vecs = model.encode(words, normalize_embeddings=True)  # (18, 384)

# Reduce to 2D for visualisation
tsne   = TSNE(n_components=2, perplexity=5, random_state=42)
coords = tsne.fit_transform(word_vecs)

colors = ["#3fb950"] * 6 + ["#e3b341"] * 6 + ["#58a6ff"] * 6

plt.figure(figsize=(10, 7))
for i, word in enumerate(words):
    plt.scatter(coords[i, 0], coords[i, 1], color=colors[i], s=100, zorder=3)
    plt.annotate(word, (coords[i, 0] + 0.3, coords[i, 1] + 0.3), fontsize=9)

plt.title("Word Embeddings in 2D (t-SNE projection)")
plt.axis("off")
plt.tight_layout()
plt.savefig("embeddings_tsne.png", dpi=150)
# Expected: three clean clusters — animals (green), royalty (yellow),
# programming (blue). Within each cluster: cat↔kitten closest, king↔queen
# adjacent, python↔java near each other.

10 — The Embedding Evolution: 1950s to 2026

11 — What Embeddings Still Can't Do

12 — A Thought to Sit With

Every word you have ever spoken lives — in some representation — inside these models.

Not as a sound, not as ink, not as memory.
As coordinates in a space of three hundred dimensions.

The question isn't whether that's impressive.
The question is: what gets lost in translation to geometry?

What parts of human language refuse to be a point in space?