Feature Engineering & Selection:
Where Domain Expertise Meets ML

In 2012, a Netflix challenge team improved recommendations by 10%
not by finding a better algorithm — by engineering a new feature:
how long a user hesitated before clicking.

A hesitation of 3 seconds before watching a romantic comedy
says something very different about a person's true preferences
than confidently clicking it immediately.
The raw data had this signal all along. Nobody had thought to extract it.

01 — Why Features Matter More Than Algorithms

You've spent 13 days learning algorithms. Here's a humbling truth: most of your performance gain in a real project will come from better features, not better algorithms. Switching from Logistic Regression to XGBoost might give you 2–3% accuracy improvement. Engineering the right features can give you 15–20%.

An algorithm can only learn what the features allow it to see. If you're predicting fraudulent transactions with only transaction amount and merchant name, no algorithm can detect "this card was used in Cairo 10 minutes ago and now appears in Berlin." The information isn't there. The algorithm is powerless. Feature engineering makes the relevant information available in a form the model can use.

Feature engineering is also where domain knowledge becomes a competitive advantage. Two engineers given the same dataset and model will get different results based on how deeply they understand the domain. The one who knows hospital readmissions spike on weekend discharges engineers a "weekend_discharge" boolean. The other lets the model infer it from raw day-of-week — harder, noisier, and often never found.

02 — The Three Phases: Create, Transform, Select

03 — Feature Creation: Code and Patterns

The five most reliable feature creation patterns — each demonstrated with real code and output:

import pandas as pd
import numpy as np

# Sample transaction dataset
df = pd.DataFrame({
    'amount':    [100, 5000, 23, 8200, 450, 12],
    'timestamp': pd.to_datetime([
        '2026-01-15 02:30', '2026-01-15 14:00',
        '2026-01-16 03:15', '2026-01-16 15:45',
        '2026-01-17 22:10', '2026-01-17 09:05']),
    'city':      ['Cairo', 'Berlin', 'Cairo', 'Cairo', 'Berlin', 'Cairo'],
    'is_fraud':  [0, 1, 0, 1, 0, 0],
})

# ── 1. Temporal decomposition ────────────────────────────────────────────
df['hour']       = df['timestamp'].dt.hour
df['day_of_week'] = df['timestamp'].dt.dayofweek   # 0=Mon, 6=Sun
df['is_night']   = (df['hour'] < 6).astype(int)   # 00:00–05:59 → 1
df['is_weekend']  = (df['day_of_week'] >= 5).astype(int)

# ── 2. Log transformation (compress right-skewed amount) ─────────────────
df['log_amount']  = np.log1p(df['amount'])   # log(x+1) handles zeros

# ── 3. Aggregation: rolling history per city ─────────────────────────────
df['city_tx_count'] = df.groupby('city')['amount'].transform('count')
df['city_avg_amt']  = df.groupby('city')['amount'].transform('mean')

# ── 4. Ratio / interaction feature ───────────────────────────────────────
df['amount_vs_city_avg'] = df['amount'] / df['city_avg_amt']
# Ratio > 1.0 means this transaction is larger than usual for this city

# ── 5. Target encoding (city → mean fraud rate for this city) ────────────
city_fraud_rate = df.groupby('city')['is_fraud'].mean()
df['city_fraud_rate'] = df['city'].map(city_fraud_rate)

cols = ['amount', 'log_amount', 'hour', 'is_night',
        'amount_vs_city_avg', 'city_fraud_rate', 'is_fraud']
print(df[cols].to_string(index=False))

04 — Feature Transformation: Distributions That Models Can Read

Algorithms have assumptions about feature distributions. Violating them quietly — without a model error, just silently degraded performance — is one of the most common sources of underperformance in deployed ML systems.

import numpy as np
from sklearn.preprocessing import StandardScaler, QuantileTransformer

# Simulate income data: right-skewed (most people earn little, few earn a lot)
np.random.seed(42)
income = np.random.exponential(scale=40000, size=1000)  # mean ~40K, max ~500K+

log_income      = np.log1p(income)                  # log(x+1)
scaled_income   = StandardScaler().fit_transform(income.reshape(-1,1)).flatten()
quantile_income = QuantileTransformer(output_distribution='normal').fit_transform(
                      income.reshape(-1,1)).flatten()

# Summary statistics show the transformation effect
print(f"{'Transform':<22} {'Mean':>8} {'Std':>8} {'Min':>10} {'Max':>10} {'Skew':>7}")
print("-" * 70)
for name, arr in [("Raw income", income),
                    ("Log(income+1)", log_income),
                    ("StandardScaler", scaled_income),
                    ("QuantileTransform", quantile_income)]:
    from scipy.stats import skew
    print(f"{name:<22} {arr.mean():>8.1f} {arr.std():>8.1f} {arr.min():>10.1f} {arr.max():>10.1f} {skew(arr):>7.2f}")

05 — Encoding Categorical Features

How you encode a categorical column is one of the most impactful decisions in a pipeline. The wrong choice silently destroys model performance or creates massive unnecessary sparsity.

06 — Target Leakage: The Most Dangerous Trap

Target leakage is when a feature contains information about the target that would not be available at prediction time. Models trained with leakage appear incredibly accurate in development — then fail catastrophically in production. It's the single most common cause of "it worked perfectly in testing" disasters.

import numpy as np
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

np.random.seed(42)
N = 1000

# Legitimate features available at prediction time
credit_score = np.random.randint(300, 850, N)
income       = np.random.exponential(50000, N)
loan_amount  = np.random.uniform(5000, 50000, N)

# True default: driven by legitimate features
default_prob = (1 - (credit_score - 300) / 550) * 0.4
y = (np.random.rand(N) < default_prob).astype(int)

# LEAKY feature: collection calls received AFTER default occurs
collection_calls = y * np.random.randint(3, 15, N)  # only non-zero if defaulted!

X_leaky = np.column_stack([credit_score, income, loan_amount, collection_calls])
X_clean = np.column_stack([credit_score, income, loan_amount])

X_tr_l, X_te_l, y_tr, y_te = train_test_split(X_leaky, y, test_size=0.2, random_state=42)
X_tr_c, X_te_c, _, _       = train_test_split(X_clean, y, test_size=0.2, random_state=42)

rf = RandomForestClassifier(n_estimators=100, random_state=42)

rf.fit(X_tr_l, y_tr)
acc_leaky_test = accuracy_score(y_te, rf.predict(X_te_l))

# Simulate production: collection_calls = 0 for everyone (loan not issued yet)
X_prod = X_te_c.copy()
X_prod_with_zero_calls = np.column_stack([X_prod, np.zeros(len(X_prod))])
acc_leaky_prod = accuracy_score(y_te, rf.predict(X_prod_with_zero_calls))

rf.fit(X_tr_c, y_tr)
acc_clean = accuracy_score(y_te, rf.predict(X_te_c))

print("Model              Test Accuracy   Production Accuracy   Gap")
print("-" * 60)
print(f"With leakage       {acc_leaky_test:.3f}           {acc_leaky_prod:.3f}                {acc_leaky_test-acc_leaky_prod:.3f}")
print(f"Without leakage    {acc_clean:.3f}           {acc_clean:.3f}                0.000")

07 — Feature Selection: Three Methods with Code

After creating and transforming features, you'll often have 50–200 candidate features. Selection decides which to keep. These three methods cover the full spectrum from fast and simple to slow and thorough:

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import cross_val_score
from sklearn.feature_selection import (SelectKBest, mutual_info_classif,
                                              RFE, SelectFromModel,
                                              VarianceThreshold)
from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegression

data = load_breast_cancer()
X, y = data.data, data.target
feature_names = data.feature_names

# ── Baseline: all 30 features ────────────────────────────────────────────
rf = RandomForestClassifier(n_estimators=100, random_state=42)
baseline_cv = cross_val_score(rf, X, y, cv=5).mean()

# ── Method 1: Filter — Mutual Information (fast, model-free) ─────────────
# Mutual information: how much knowing X reduces uncertainty about y
mi_selector = SelectKBest(mutual_info_classif, k=10)
X_mi = mi_selector.fit_transform(X, y)
mi_cv = cross_val_score(rf, X_mi, y, cv=5).mean()
mi_features = feature_names[mi_selector.get_support()]

# ── Method 2: Wrapper — RFE (slower, captures interactions) ──────────────
rfe = RFE(RandomForestClassifier(n_estimators=50, random_state=42), n_features_to_select=10)
X_rfe = rfe.fit_transform(X, y)
rfe_cv = cross_val_score(rf, X_rfe, y, cv=5).mean()
rfe_features = feature_names[rfe.support_]

# ── Method 3: Embedded — RF importance threshold ─────────────────────────
emb_selector = SelectFromModel(rf.fit(X, y), threshold='mean')
X_emb = emb_selector.transform(X)
emb_cv = cross_val_score(rf, X_emb, y, cv=5).mean()

print(f"{'Method':<28} {'Features':>9} {'CV Accuracy':>12}")
print("-" * 53)
print(f"{'Baseline (all 30)':<28} {'30':>9} {baseline_cv:>12.4f}")
print(f"{'Filter: Mutual Info (top 10)':<28} {'10':>9} {mi_cv:>12.4f}")
print(f"{'Wrapper: RFE (top 10)':<28} {'10':>9} {rfe_cv:>12.4f}")
print(f"{'Embedded: RF importance':<28} {X_emb.shape[1]:>9} {emb_cv:>12.4f}")

print(f"\nFilter top 10 features:\n  {list(mi_features)}")
print(f"\nRFE top 10 features:\n  {list(rfe_features)}")

08 — The Before/After: Does Engineering Actually Help?

All the theory is worth nothing if the numbers don't move. Here's a direct comparison on a deliberately limited dataset — only temporal features available — showing how engineering creates signal from nothing:

import numpy as np
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.model_selection import cross_val_score

np.random.seed(42)
N = 2000

# Simulate fraud detection with timestamp + amount only
hours   = np.random.randint(0, 24, N)
amounts = np.random.exponential(200, N)

# True fraud pattern: night + high amount → more fraud
fraud_prob = (0.05
    + 0.25 * (hours < 6).astype(float)       # night → +25%
    + 0.20 * (amounts > 500).astype(float))   # high amount → +20%
y = (np.random.rand(N) < fraud_prob.clip(0, 1)).astype(int)

# RAW features: hour (0–23) and raw amount
X_raw = np.column_stack([hours, amounts])

# ENGINEERED features: extract the exact patterns that generate fraud
is_night    = (hours < 6).astype(float)
log_amount  = np.log1p(amounts)
high_amount = (amounts > 500).astype(float)
X_eng = np.column_stack([hours, amounts, log_amount, is_night, high_amount])

gb = GradientBoostingClassifier(n_estimators=100, random_state=42)
cv_raw = cross_val_score(gb, X_raw, y, cv=5, scoring='roc_auc')
cv_eng = cross_val_score(gb, X_eng, y, cv=5, scoring='roc_auc')

print(f"{'Features':<20} {'AUC Mean':>10} {'AUC Std':>9}")
print("-" * 42)
print(f"{'Raw (2 features)':<20} {cv_raw.mean():>10.4f} {cv_raw.std():>9.4f}")
print(f"{'Engineered (5 feat.)':<20} {cv_eng.mean():>10.4f} {cv_eng.std():>9.4f}")
print(f"\nAUC improvement: +{cv_eng.mean()-cv_raw.mean():.4f} ({(cv_eng.mean()-cv_raw.mean())/cv_raw.mean()*100:.1f}%)")

09 — When It Matters Most

10 — A Thought to Sit With

11 — Coming Up Next

Day 15 — CNNs from Scratch. We close the classical ML phase and enter deep learning. Convolutional Neural Networks are what made computer vision tractable — understanding them from first principles (convolutions, pooling, receptive fields) gives you the mental model for every vision architecture that came after, including the Vision Transformers that now power modern image models.

12 — Go Deeper

🔗 Kaggle Feature Engineering Course
Free · Hands-on notebooks covering mutual information, target encoding, and principal components.

🔗 Feature Engineering Full Course
In this 1-hour tutorial, I'll guide you through the ins and outs of one of the most critical steps in data science and machine learning. Whether you're a beginner or an experienced data scientist, this course will equip you with the tools and techniques you need to effectively transform raw data into features that can be used to train machine learning models.

📄 Why do tree-based models still outperform deep learning on tabular data? — Grinsztajn et al. (2022)
Shows that feature preprocessing and engineering explain much of the tree vs neural network gap on tabular benchmarks. Directly relevant to why everything in this post matters.