RAG: Retrieval-Augmented
Generation Pipeline

You wouldn't expect a brilliant colleague to know the contents
of a document they've never read.
But you would expect them to read it quickly if you handed it to them,
then answer your questions accurately based on what it says.

RAG is how you hand documents to a language model.

01 — The Problem RAG Solves

Every LLM has a knowledge cutoff — a date after which it knows nothing. GPT-4's cutoff is early 2024. LLaMA-3's is early 2024. Whatever happened after that date, whatever lives in your private documents, whatever is in your company's internal wiki — the model has never seen any of it. Ask about it and it will either say "I don't know" or, worse, invent something plausible-sounding that is completely wrong.

The obvious solution is to put the relevant information in the prompt. If you want the model to answer questions about a specific document, just paste the document in. For a short document, this works fine. But what if you have 500 documents? Or 10,000? You can't paste them all in — most models have a context window of 8k–128k tokens, and even a moderate knowledge base blows past that immediately. And even if you could fit everything in, it would be incredibly slow and expensive to process the entire library on every single question.

RAG solves three problems at once:

02 — The RAG Pipeline at a Glance

RAG has two distinct phases. The first happens once, offline, before any questions arrive. The second happens on every single query in real time.

That's the whole pipeline. Every RAG system — from a quick weekend project to a production enterprise search engine — follows this exact shape. The details differ; the structure doesn't.

03 — Stage by Stage: What Each Component Does

Stage 1 — Load: Getting Text Into the Pipeline

The first step is simply reading your documents. PDFs, Word files, web pages, Markdown files, database exports — everything needs to be converted to plain text before anything else can happen. Libraries like LangChain provide loaders for almost every format. PyMuPDF handles PDFs reliably. Unstructured extracts clean text from mixed-format documents including tables and headers.

The goal of this stage is simple: get the text out. Clean it up — remove page headers, footers, duplicate whitespace — and you're ready for the next step.

Stage 2 — Chunk: Splitting Into Manageable Pieces

You cannot embed an entire document as a single unit. If a user asks about one specific policy in a 50-page handbook, you don't want to retrieve all 50 pages — you want to retrieve the 3 paragraphs that actually answer the question. Chunking splits your documents into smaller, self-contained pieces that can be individually retrieved.

The most common approach is recursive character chunking: split at paragraph breaks first, then at sentence breaks, then at word breaks — only splitting smaller if the chunk still exceeds the target size. A typical chunk size for RAG is 300–500 tokens with a 50–100 token overlap between adjacent chunks, so that a sentence split across a boundary doesn't lose its context.

Stage 3 — Embed: Turning Text into Numbers

This is where Day 23 (Word Embeddings) comes back in full force. Once you have your chunks, you convert each one into a vector — a list of numbers that represents its meaning. The embedding model is trained so that similar text produces similar vectors. Two chunks that talk about "refund policy" will have vectors that point in nearly the same direction in the high-dimensional space, even if they use slightly different words.

Popular embedding models: BGE-M3 and E5-large (open-source, excellent quality), Sentence-BERT (lightweight, fast), OpenAI text-embedding-3-small (cloud API, easy to use). All of these are BERT-style encoders — exactly the architecture from Day 26 — producing a fixed-size vector from the mean-pooled token outputs.

Stage 4 — Store: The Vector Database

The embeddings are stored in a vector database — a specialised database that is optimised for one specific operation: given a query vector, find the stored vectors that are most similar to it. This is called Approximate Nearest Neighbour (ANN) search, and it's what makes retrieval fast even across millions of chunks.

For a first project, FAISS or Chroma will take you further than you think. Move to a managed service like Pinecone only when you have millions of documents and need guaranteed uptime.

Stage 5 — Retrieve: Finding What's Relevant

When a question arrives, you embed it using the same embedding model used to embed the chunks. The resulting query vector is then compared to every stored chunk vector using cosine similarity — the same metric from Day 23. The chunks with the highest similarity scores are the most relevant ones and get retrieved.

The beauty of this connection: it's the exact same vector similarity concept from Day 23, just applied at scale. "king − man + woman ≈ queen" used the same cosine distance to find the closest vector. Here, "What is the refund policy?" finds the chunk whose vector is closest to the query vector. Same math, same intuition, production use case.

Stage 6 — Generate: The Grounded Answer

The retrieved chunks are assembled into a prompt — along with the original question and a clear instruction to answer only based on the provided context. This prompt is sent to an LLM (GPT-4, LLaMA-3, Claude — whatever you're using) which generates the final answer.

04 — The Simplest Possible Working RAG System

Below is the simplest end-to-end RAG pipeline that actually works — no LangChain abstractions, no external APIs required. Pure Python with two libraries: sentence-transformers for embeddings and faiss-cpu for vector search. This is the clearest way to see exactly what's happening at each step.

# pip install sentence-transformers faiss-cpu
# That's all you need. No LangChain, no API keys, no cloud service.

import numpy as np
import faiss
from sentence_transformers import SentenceTransformer

# ── Your "knowledge base" (in a real system: loaded from PDFs/docs) ──────
documents = [
    "Refunds are processed within 5 business days of the approved request.",
    "To request a refund, email support@company.com with your order number.",
    "Refunds are not available for digital downloads or subscription plans.",
    "Shipping typically takes 3-7 business days within the country.",
    "Express shipping takes 1-2 days and costs an additional $15.",
    "Our office hours are Monday to Friday, 9am to 6pm.",
    "You can track your order using the link sent to your email.",
]

# ── Stage 1: Embed all documents (done once, offline) ────────────────────
model = SentenceTransformer("all-MiniLM-L6-v2")   # small, fast, free
doc_embeddings = model.encode(documents)            # shape: (7, 384)

# ── Stage 2: Build the vector index ──────────────────────────────────────
dim = doc_embeddings.shape[1]                       # 384 dimensions
index = faiss.IndexFlatL2(dim)                      # exact nearest-neighbour search
index.add(doc_embeddings.astype(np.float32))        # store all vectors
print(f"Index built: {index.ntotal} documents stored\n")

# ── Stage 3: Answer a question ───────────────────────────────────────────
def ask(question: str, k: int = 3) -> str:
    """Retrieve top-k relevant chunks and build a grounded answer."""

    # Embed the question using the SAME model
    q_vec = model.encode([question]).astype(np.float32)

    # Find k most similar document vectors
    distances, indices = index.search(q_vec, k)

    # Collect the retrieved chunks
    retrieved = [documents[i] for i in indices[0]]

    # Build a simple grounded answer (in production: send to an LLM)
    context = "\n".join([f"  • {c}" for c in retrieved])
    return f"Question: {question}\n\nRelevant context found:\n{context}"


# Try it out
print(ask("How long does a refund take?"))
print("\n" + "─"*55 + "\n")
print(ask("Can I track my delivery?"))
print("\n" + "─"*55 + "\n")
print(ask("Is express delivery available?"))

05 — A Real Use Case: The Personal AI Assistant

Let's make this concrete with the simplest possible real-world scenario. You have a folder of documents about yourself — your CV, portfolio projects, publications, a personal bio. You want to build a chatbot that can answer questions about you accurately, without hallucinating.

06 — The Connection to Word Embeddings (Day 23)

RAG is not a new idea — it's the same vector similarity from Day 23, applied at scale with a purpose. Let's trace the exact connection:

The only difference between "find the word closest to 'king − man + woman'" and "find the document chunk closest to 'What is the refund policy?'" is the scale and the purpose. The underlying operation — cosine similarity in an embedding space — is identical. This is why understanding word embeddings first makes RAG immediately intuitive: you've already seen how meaning maps to geometry. RAG just applies that geometry to search.

07 — RAG vs Fine-Tuning: Which One Do You Need?

After Day 28 (Fine-Tuning) and Day 29 (RAG), you now know both major approaches to making a model domain-specific. They are not alternatives — they serve fundamentally different purposes and are often used together.

08 — What Makes a RAG System Fail (and How to Fix It)

09 — A Thought to Sit With

10 — Coming Up Next

Tomorrow — Day 30: Prompt Engineering. You now know how to build systems that retrieve the right information. Day 30 covers how to talk to the model once you have it: zero-shot and few-shot prompting, chain-of-thought, system prompts, and the full picture of how prompt design affects output quality — including what can go wrong when prompts are manipulated.

11 — Go Deeper

📄 Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks — Lewis et al. (2020)
The original RAG paper from Facebook AI. Introduced the term and the architecture. Section 2 (model) and Section 4 (results) are the essential parts.

🔗 LangChain RAG Tutorial — Official Docs
The clearest step-by-step tutorial for building a production RAG system with LangChain, including document loaders, text splitters, embeddings, and retrieval chains.

🔗 Chroma — Getting Started
The easiest vector database to start with. Runs locally, integrates with LangChain and LlamaIndex out of the box. The quickstart takes under 10 minutes.

📄 RAGAS: Automated Evaluation of RAG — Es et al. (2023)
The evaluation framework for RAG systems — faithfulness, answer relevancy, context precision. Essential reading before deploying a RAG system in production.

▶ RAG from Scratch — LangChain (2024)
YouTube series building RAG from first principles in short focused videos. The best hands-on companion to this post.