A chef doesn't walk into a kitchen and immediately serve a dish.
They source ingredients, clean them, prep them, cook them, taste and adjust,
then plate and serve. If any step is skipped, the dish fails — regardless
of how talented the chef is.
Machine learning has the exact same workflow. It's called the ML pipeline.
01 — Why the Pipeline Matters More Than the Model
New practitioners obsess over model choice: should I use a Random Forest or XGBoost? Should I try a neural network? Experienced engineers know a different truth: the model is almost never the bottleneck.
In practice, 80% of a data scientist's time is spent on data — finding it, cleaning it, understanding it, transforming it. A simple model running on clean, well-prepared data almost always outperforms a sophisticated model running on messy, raw data.
Understanding the pipeline is understanding where ML projects actually succeed or fail.
Think of building a house. You could hire the world's best architect — but if the foundation is poorly laid and the plumbing runs backwards, the beautiful design is worthless. Data is your foundation. The model is the architecture. Skip the foundation and the house falls.
02 — The Seven Stages
Before touching any data or code, be ruthlessly clear: What decision do you want the model to make? Is it predicting a number (regression) or assigning a category (classification)? What does success look like — accuracy? Revenue? Lives saved? A vague problem always produces a useless model.
Gather raw data relevant to the problem. Sources include databases, APIs, web scraping, sensors, or human annotation. Quantity and quality both matter — but quality wins. One thousand accurate, representative examples beat ten thousand noisy, biased ones every time.
Real-world data is messy: missing values, duplicate rows, typos, impossible entries (a person aged 847), inconsistent formatting. Exploratory Data Analysis (EDA) means plotting distributions, finding correlations, and spotting outliers so you understand what you're actually working with before you model it.
Raw data rarely feeds directly into a model. You transform, combine, and create features — the input variables the model learns from. For example: turning a birth date into "age," encoding a city name as a number, or extracting the day-of-week from a timestamp. This is where domain expertise becomes a genuine superpower.
Your data is divided into three non-overlapping buckets. The training set is what the model learns from. The validation set is used to tune the model's settings — its hyperparameters (configuration choices set before training, like how deep a decision tree grows or how fast it learns). The test set is held back until the very end to give a final, honest estimate of real-world performance.
Choose a model, train it on the training set, then evaluate it on the validation set using metrics suited to the problem — accuracy for balanced categories, F1 score for imbalanced ones (like fraud or disease), RMSE for numeric predictions. Tune the hyperparameters and repeat until the results are satisfying.
The trained model is packaged (often with Docker), exposed via an API (Flask, FastAPI), and integrated into a product. But this isn't the end. Models degrade as the world changes.
Data drift — the inputs your model receives start to differ from what it was trained on.
Model drift — its predictions become less accurate as a result.
Both require ongoing monitoring.
03 — The Critical Concept: Data Leakage
Data leakage is accidentally letting the model see information it wouldn't have access to at prediction time. It makes your model look brilliant during evaluation — and fail completely once deployed.
Imagine you're building a model to predict whether a student will pass their final exam — based only on information available at the start of the semester. If you accidentally include their final exam score as one of the input features, the model achieves 99% accuracy in testing, because the answer is literally sitting in the data. In the real world, that score doesn't exist yet when you need to make the prediction — so the model is completely useless.
The rule: your model must only see information it would realistically have at the exact moment it needs to make a prediction.
04 — A Correct Pipeline in Python
Here's a complete, properly structured pipeline on a real medical dataset — 569 tumors classified as benign or malignant. Every line reflects something discussed above.
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report
# Stage 2 — Load data (569 real tumor records: benign vs malignant)
X, y = load_breast_cancer(return_X_y=True)
# Stage 5 — Three-way split: 60% train | 20% validation | 20% test
X_train, X_temp, y_train, y_temp = train_test_split(
X, y, test_size=0.4, random_state=42, stratify=y
)
X_val, X_test, y_val, y_test = train_test_split(
X_temp, y_temp, test_size=0.5, random_state=42, stratify=y_temp
)
# → 341 train | 114 validation | 114 test
# Stage 4 — Feature engineering: scale to zero mean, unit variance
# CRITICAL: fit only on training data — never expose test statistics here
scaler = StandardScaler()
X_train = scaler.fit_transform(X_train) # learn mean & std from training data only
X_val = scaler.transform(X_val) # apply same scale — no peeking at test
X_test = scaler.transform(X_test) # apply same scale — no peeking at test
# Stage 6 — Train the model
model = LogisticRegression(max_iter=200)
model.fit(X_train, y_train)
# Stage 6 — Tune using validation set (adjust hyperparameters here, repeat)
val_score = model.score(X_val, y_val)
# → ~97% — if satisfied, proceed; otherwise adjust max_iter, C, etc.
# Stage 6 — Final honest evaluation on the test set (touch only once)
print(classification_report(y_test, model.predict(X_test)))Notice: the scaler is fit only on training data, then transformed onto everything else. Fitting on the full dataset before splitting would leak test statistics into the scaler — a subtle but important form of data leakage.
| Metric | What it measures | Best used when… |
|---|---|---|
| Accuracy | % of all predictions that are correct | Classes are roughly balanced in size |
| F1 Score | Balance between catching true cases (recall) and avoiding false alarms (precision) | One class is rare — fraud, disease, spam |
| RMSE | Typical prediction error in the original units (e.g. ±$4,000 for house prices) | Predicting a continuous number, not a category |
The code above produces output like this — read it line by line:
05 — Why This Matters in Production
Three principles that separate engineers who ship reliable ML from those who don't:
- The pipeline is the product. In production, you're not shipping a model — you're shipping the entire pipeline: preprocessing, model, post-processing. If any step breaks or drifts, the product breaks.
- Reproducibility is non-negotiable. Set random seeds. Version your data. Log your hyperparameters. If you can't reproduce your results, you can't debug them — and you can't trust them.
- The test set is sacred. Look at it once, at the very end. Every time you peek at it to make a decision, you're leaking information and your final accuracy number becomes a lie.
Training GPT-4 follows this exact pipeline — just at a scale requiring thousands of GPUs and months of continuous compute. The data cleaning step alone (filtering the entire internet for quality text) is one of the most important and least-discussed reasons why modern LLMs work well. The pipeline is universal. Only the scale changes. The principles never do.
transform() on everything else. The test set must behave like completely unknown future data.- Problem Definition — Clarify what the model must decide, and what success actually looks like.
- Data Collection — Gather high-quality, representative data from databases, APIs, sensors, or annotations.
- Data Cleaning & EDA — Fix errors, handle missing values, explore distributions. This takes most of your time.
- Feature Engineering — Transform raw data into the meaningful inputs a model can learn from.
- Train / Val / Test Split — Three separate buckets: learn, tune, then evaluate honestly — in that order.
- Model Training & Evaluation — Train the model, measure with the right metric, tune hyperparameters, repeat.
- Deployment & Monitoring — Ship it, then watch it continuously. Models degrade silently in the real world.
06 — A Thought to Sit With
If a model is trained on historical data, it learns historical biases. A hiring model trained on past successful employees might learn to favour candidates who resemble those past employees — quietly encoding discrimination into mathematics.
Who is responsible for the pipeline's fairness — the data collector, the engineer, or the business deploying it?
07 — Coming Up Next
Tomorrow — Day 3: How Models Actually Learn. We open the black box of training itself: what a loss function is, how gradient descent works (the engine behind every model in existence), and why the learning rate is the most consequential number you'll ever tune.
Imagine you're blindfolded on a hilly landscape, trying to reach the lowest valley. You can only feel the slope under your feet — so you take a small step in whichever direction feels downhill, over and over. That's gradient descent. Tomorrow you'll understand exactly why this blind, incremental process is how every neural network, every language model, and every recommendation system gets smarter.
08 — Go Deeper
▶ IBM: What is a Data Pipeline?
YouTube · 9 min · A clear walkthrough of the full pipeline — great starting point, no prior knowledge needed.
▶ Kaggle: Intro to EDA in Python
Interactive notebook · Run it live in your browser · Covers exactly the EDA stage from this article — missing values, distributions, correlations, and outliers with real code.
▶ scikit-learn: Data Preprocessing Guide
Official docs · Hands-on · A detailed reference for the cleaning and feature engineering stages — scaling, encoding, and handling missing data.
▶ Kaggle: Intro to Machine Learning
Free course · Write and run real code in your browser · Covers the full pipeline end-to-end and pairs well with this series.
- What is data leakage and why is it dangerous?
Data leakage occurs when information from outside the training set improperly influences the model — making evaluation results look better than real-world performance will be. It produces models that appear excellent in testing but fail in production. - Why do we need three splits (train / validation / test) instead of two?
The validation set is used for model selection and hyperparameter tuning. Each tuning decision based on it leaks a small amount of information. The test set is kept completely separate to give a final, unbiased performance estimate that no prior decision has touched. - Why must the scaler be fit only on training data?
Fitting on the full dataset lets the scaler see test data statistics (mean, standard deviation), which is data leakage. The test set must be treated as completely unseen future data — as if it arrived after the model was already trained. - What is EDA and why does it matter?
Exploratory Data Analysis is the process of visualising and summarising data to understand its structure, spot anomalies, and uncover patterns before modelling. Skipping EDA means building models on data you don't actually understand. - What is the difference between a model parameter and a hyperparameter?
Parameters are learned automatically from training data — the weights in a neural network or the coefficients in a regression. Hyperparameters are configuration choices set by the engineer before training begins, such as learning rate, tree depth, or number of neighbours. The model learns parameters; you tune hyperparameters.
No attachments for this topic yet.