A surgeon performing robotic-assisted surgery needs to know
not just that an artery is present in the image —
but exactly where it is, pixel by pixel,
so the robotic arm doesn't nick it.
A self-driving car's computer needs to know not just
that a pedestrian is in the scene —
but their precise bounding box, updated 30 times per second.
Classification was the beginning. Localisation is the requirement.
01 — Three Tasks, Three Levels of Spatial Understanding
Computer vision has three fundamental output types. Each requires a different output format, a different loss function, and a different amount of annotation effort to train. Understanding the hierarchy makes the whole field make sense.
Image classification (Days 15–17) produces a single label for the entire image. Input: 224×224 pixels. Output: "cat" (probability 0.97). The spatial location of the cat is irrelevant — the label is global. One forward pass, one vector, one argmax.
Object detection produces a label and a bounding box for each object present. Output: [("person", x=120, y=80, w=60, h=140, conf=0.91), ("car", x=340, y=200, w=180, h=90, conf=0.87)]. The model must simultaneously classify and localise — two tasks in one network, with a loss function that penalises both wrong labels and imprecise boxes.
Segmentation assigns a class label to every single pixel. Semantic segmentation gives each pixel a class ("road," "sky," "person") without distinguishing between individuals. Instance segmentation goes further: it distinguishes the pedestrian on the left from the pedestrian on the right, giving each a separate per-pixel mask. Panoptic segmentation combines both.
Classification is like answering "what country is this city in?" Detection is like drawing labelled rectangles around every building on a map. Semantic segmentation is like colour-coding every square metre — residential red, commercial blue, park green — without naming individual buildings. Instance segmentation is that same colour-coded map but also numbering each individual building separately so you can refer to "Building 47."
Each level requires more annotation effort, more compute, and unlocks different downstream applications. A tumour detector needs bounding boxes; a surgical robot needs pixel masks.
| Task | Output | Loss | Annotation cost | Typical use |
|---|---|---|---|---|
| Classification | Class label + probability | Cross-entropy | ✓ Low — one label/image | Spam filter, disease screening |
| Detection | Boxes + labels + confidence | Loc + obj + cls loss | ~ Medium — boxes per object | Autonomous vehicles, surveillance |
| Segmentation | Per-pixel class mask | Pixel-wise cross-entropy / Dice | ✗ High — every pixel labelled | Medical imaging, robotics |
02 — The Architecture Timeline: How We Got Here
Computer vision didn't leap from "no CNN" to "YOLOv8 detects 80 classes at 160 FPS." It evolved through a chain of papers, each fixing a specific bottleneck in the previous one. Understanding the lineage tells you why each architecture looks the way it does — and which problems are still open.
Three parallel tracks evolved simultaneously: the backbone CNNs that extract features (shared across all tasks), the detection architectures built on top of them, and the segmentation architectures that repurposed the same backbones for pixel-level output.
F(x) + x). Gradients now have a direct highway back. ResNet-50 (25M params) achieves lower error than VGG-19 (138M params). ResNet backbones are the default starting point for detection and segmentation today.Do Detection and Segmentation Use the Same Backbone CNNs?
This is a crucial practical question. The short answer: yes, the backbone is almost always shared — but the head is task-specific. The backbone's job is feature extraction, which is the same regardless of what the features are used for. Only the output head (what you do with those features) changes per task.
yolov8s.pt, its backbone is a modified CSP-ResNet. When you load a U-Net for medical imaging, you typically replace its encoder with a pretrained ResNet-50. The same weights that classify ImageNet dogs also extract edge and texture features useful for detecting pedestrians and segmenting tumours.YOLO('yolov8s.pt') → replace backbone with ConvNeXt → same head, different features.| Situation | Recommended architecture | Why |
|---|---|---|
| Real-time detection (>30 FPS) | YOLOv8n / YOLOv8s | Single-pass, anchor-free, GPU-optimised |
| Highest accuracy detection (offline) | DINO / Faster R-CNN + ResNet-101 | Two-stage precision on small/dense objects |
| Medical image segmentation | nnU-Net (auto-configured) | Best-in-class on 23/23 benchmarks, no tuning |
| Natural scene segmentation | DeepLabv3+ or Mask R-CNN | ASPP captures multi-scale context |
| Zero-shot / prompt-based | SAM2 | Universal, no fine-tuning needed |
| Limited data (<1,000 images) | EfficientNet-B3 backbone + transfer | Compound scaling, best accuracy/param ratio |
| Mobile / edge deployment | YOLOv8n or MobileNetV3 + SSD | Sub-5ms inference on embedded hardware |
03 — How Object Detection Works: From R-CNN to YOLO
The fundamental challenge in detection: you don't know in advance how many objects are in an image, where they are, or how large they are. You need a network that outputs a variable-length list of (box, class, confidence) tuples from a fixed-size input.
The classic two-stage approach — the R-CNN family — solved this in two steps. First, a region proposal network identifies ~2,000 candidate patches that might contain objects. Second, a CNN classifies each patch. R-CNN (2014) was accurate but processed one image in 47 seconds. Fast R-CNN got this to seconds by sharing convolutional features. Faster R-CNN got to ~200ms by making the region proposal network part of the same network. But "two stages, two passes" had a ceiling on speed that no engineering could overcome.
YOLO (You Only Look Once, 2015) took a fundamentally different approach: frame detection as a single regression problem. Divide the image into a grid. Have each grid cell directly predict boxes and class labels simultaneously. One network, one pass, all detections at once. The original YOLO ran at 45 FPS — real-time detection for the first time. YOLOv8 (2023) exceeds 160 FPS on modern hardware.
Two-stage detectors (Faster R-CNN, Mask R-CNN) remain more accurate on small objects and dense scenes because their region proposals are more precise. YOLO-family one-stage detectors are faster because they skip the proposal step entirely. For real-time applications (autonomous driving, live video), YOLO wins. For offline analysis of medical images where every pixel matters, two-stage or segmentation models win. Modern benchmarks on COCO: YOLOv8x reaches ~53 mAP; DINO (transformer-based two-stage) reaches ~63 mAP, at ~10× the inference cost.
04 — Inside YOLO: How One Forward Pass Detects Everything
YOLO's architecture has three main components: a backbone (feature extraction), a neck (multi-scale feature fusion), and a head (final predictions). Here's how each works:
Intersection over Union (IoU) = area of overlap between predicted box and ground-truth box ÷ area of their union. A perfect prediction has IoU = 1.0. Complete miss has IoU = 0. IoU > 0.5 is the conventional threshold for a "correct" detection in standard benchmarks. IoU appears in three places in YOLO: the loss function (penalise imprecise boxes), confidence thresholding (is there actually an object here?), and NMS (are these two boxes covering the same object?).
The COCO benchmark evaluates mAP at IoU thresholds from 0.5 to 0.95 in 0.05 steps and averages them — written as AP@[.5:.95]. This is much harder than the older PASCAL VOC metric, which is why COCO scores look lower.
05 — The Detection Loss Function
Object detection training minimises a composite loss with three independent components, each penalising a different type of error:
The three losses are summed with weights: total_loss = λ_box × L_box + λ_obj × L_obj + λ_cls × L_cls. In YOLOv8, default weights are approximately box=7.5, obj=0.5, cls=0.5. These ratios reflect that box regression is the hardest sub-task and deserves the most training signal. Getting these weights wrong (e.g., obj loss too high) causes the model to predict "something is here" confidently but fail to classify or localise it correctly.
06 — Code: Running and Fine-Tuning YOLOv8
The Ultralytics library makes YOLOv8 inference and fine-tuning remarkably concise. Below is a complete pipeline from off-the-shelf inference to fine-tuning on a custom dataset, with all the details that matter in practice.
# pip install ultralytics
from ultralytics import YOLO
import cv2, torch
# ── Model variants ────────────────────────────────────────────────────────
# yolov8n: nano — 3.2M params, ~160 FPS (embedded/mobile)
# yolov8s: small — 11M params, ~120 FPS
# yolov8m: med — 25M params, ~80 FPS
# yolov8l: large — 43M params, ~50 FPS
# yolov8x: xlarge— 68M params, ~30 FPS, best accuracy
model = YOLO('yolov8n.pt') # downloads pretrained weights automatically
# ── Inference on a single image ──────────────────────────────────────────
results = model('street.jpg', conf=0.4, iou=0.45)
for r in results:
for box in r.boxes:
cls_name = model.names[int(box.cls)]
conf = box.conf.item()
x1, y1, x2, y2 = box.xyxy[0].tolist()
print(f"{cls_name:12s} {conf:.2f} [{x1:.0f},{y1:.0f} → {x2:.0f},{y2:.0f}]")
# Draw boxes and save
annotated = r.plot()
cv2.imwrite('output.jpg', annotated)
# ── Fine-tune on a custom dataset ─────────────────────────────────────────
# dataset.yaml must define:
# path: /data/my_dataset
# train: images/train # folder with images + labels/*.txt (YOLO format)
# val: images/val
# nc: 3 # number of classes
# names: ['car', 'person', 'cyclist']
model = YOLO('yolov8s.pt') # start from pretrained (transfer learning)
results = model.train(
data = 'dataset.yaml',
epochs = 50,
imgsz = 640,
batch = 16,
lr0 = 0.01, # initial LR — decays via cosine schedule
patience= 10, # early stopping: stop if no improvement for 10 epochs
device = 0, # GPU 0; use 'cpu' if no GPU
augment = True, # mosaic, mixup, hsv jitter, flips — all on by default
name = 'my_detector'
)
# ── Evaluate on test set ─────────────────────────────────────────────────
metrics = model.val(data='dataset.yaml', split='test')
print(f"mAP50: {metrics.box.map50:.3f}")
print(f"mAP50-95: {metrics.box.map:.3f}")person 0.91 [120,80 → 180,220]
person 0.87 [340,75 → 395,210]
car 0.84 [50,180 → 290,280]
bicycle 0.76 [430,160 → 530,260]
traffic light 0.69 [280,20 → 310,90]
── Fine-tuning (50 epochs on custom 3-class dataset) ──────────────
epoch 10/50 box_loss=1.842 cls_loss=1.204 mAP50=0.621
epoch 25/50 box_loss=1.341 cls_loss=0.891 mAP50=0.754
epoch 50/50 box_loss=1.102 cls_loss=0.743 mAP50=0.812
── Test evaluation ────────────────────────────────────────────────
mAP50: 0.812
mAP50-95: 0.547
07 — Segmentation: Pixel-Level Understanding
Segmentation networks must output a prediction for every pixel — so the output must match the input's spatial dimensions. The standard architecture is an encoder-decoder: the encoder (a CNN or transformer backbone) compresses the image into a compact high-level representation, and the decoder gradually upsamples it back to full resolution, predicting a class label at each pixel along the way.
The problem with naive upsampling: the encoder's compression destroys spatial detail. Knowing "there is a tumour somewhere" doesn't tell you exactly where each boundary pixel falls. This is the problem U-Net (Ronneberger et al., 2015) solved.
U-Net introduced skip connections between matching encoder and decoder layers. At each resolution level, the decoder receives two inputs: the upsampled features from the level below (semantic understanding — what the object is) and the high-resolution features from the corresponding encoder layer (spatial precision — exactly where each pixel boundary falls). By concatenating these, the decoder can produce sharp boundaries that neither pure downsampling nor pure upsampling could achieve alone.
The "U" in U-Net refers to the architecture's shape: encoder contracts left-to-right (down the left side of the U), skip connections cross horizontally, decoder expands right-to-left (down the right side). U-Net and its descendants (nnU-Net, Swin-UNet, MedSAM) remain the backbone of clinical medical image analysis worldwide.
YOLO('yolov8n-seg.pt') — add -seg suffix.08 — Metrics: How You Know If It's Working
Detection and segmentation use different metrics because their outputs differ. Detection boxes are continuous (you can be "close" to the right box); segmentation masks are binary per-pixel (you either got the pixel right or you didn't). The metrics reflect this.
| Metric | Formula | Used for | Notes |
|---|---|---|---|
| IoU / Jaccard | |A ∩ B| / |A ∪ B| | Detection & segmentation | 0 = no overlap, 1 = perfect. Threshold 0.5 standard for detection "hit" |
| Dice / F1 | 2|A ∩ B| / (|A| + |B|) | Segmentation (esp. medical) | More sensitive to small regions than IoU. Preferred for small lesions, thin structures |
| AP@50 | Area under PR curve at IoU=0.5 | Detection (PASCAL VOC style) | Older, easier benchmark. 80%+ on VOC is competitive |
| mAP@[.5:.95] | Mean AP over IoU 0.5→0.95 | Detection (COCO style) | Harder, more comprehensive. 50+ is competitive on COCO 80-class |
| FPS | Frames per second | Real-time applications | >30 FPS for real-time. 5–10 FPS for offline/batch analysis is acceptable |
Consider a 512×512 MRI slice where a small lesion occupies 200 pixels out of 262,144 total. A model that predicts "no lesion anywhere" gets: IoU = 0/200 = 0 (correct — no overlap), but pixel accuracy = 262,144−200/262,144 = 99.9% (misleadingly perfect). Dice explicitly penalises missing small regions: if you miss 200 lesion pixels, Dice = 0. For medical imaging where small structures are clinically critical, Dice is the right metric. Never report pixel accuracy alone on imbalanced segmentation tasks.
09 — Why These Architectures Still Matter
These aren't dated techniques from a previous era of deep learning. They're canonical patterns that appear inside the systems you interact with daily.
Autonomous vehicles cannot function without real-time detection. A car doing 100 km/h needs to detect and localise pedestrians, cyclists, and vehicles in under 33 milliseconds per frame. YOLO's single-pass architecture makes this possible on embedded hardware. Tesla's Autopilot, Waymo's driver, and every production AV system runs detection networks in this family.
Medical imaging depends on segmentation. Tumour volume measurement, organ boundary delineation, lesion tracking over time — all require pixel-level masks, not bounding boxes. U-Net and its descendants (nnU-Net, Swin-UNet, MedSAM) are the backbone of clinical analysis pipelines in radiology and pathology worldwide. The diabetic retinopathy project from Day 17 uses segmentation as part of its grading pipeline.
The U-Net architecture lives inside generative AI. Stable Diffusion's denoising network is literally a U-Net. The encoder-decoder structure with skip connections that Ronneberger designed for medical image segmentation in 2015 is the same architecture diffusing latent representations in every image-generation model. When you generate an image with Midjourney or DALL-E, the diffusion step runs a U-Net.
Instance segmentation powers generative AI evaluation. When diffusion models generate images, quality evaluation uses segmentation models to check whether generated objects have physically plausible boundaries and spatial relationships. Detection and segmentation are embedded in the evaluation pipelines of the models you use daily.
The Segment Anything Model (SAM, Meta 2023) generalised segmentation into a foundation model: trained on 1 billion masks, it can segment any object in any image given a point, a box, or a text prompt — zero-shot, no fine-tuning. SAM2 extended this to video with a memory mechanism that propagates masks across frames. This is the vision equivalent of GPT: a single large model replacing many specialised segmentation models. SAM's architecture (Vision Transformer encoder + prompt encoder + mask decoder) uses the same attention mechanism you'll meet in Day 20 — vision and language have largely converged on the same fundamental building block.
10 — A Thought to Sit With
Object detection models can identify and track individuals in video at 30+ FPS — their face, their gait, their location, in real time. This exact capability is simultaneously the core of medical diagnostic tools that detect cancer early and save lives, and the core of mass surveillance systems that have been used to track political dissidents, religious minorities, and protesters at scale.
The same YOLOv8 weights. The same IoU calculations. The same NMS algorithm.
Can a technology be ethically neutral if its capabilities are identical regardless of application? When a dual-use tool is misused, who bears responsibility — the researchers who published the paper, the companies who built the library, the governments who deployed it, or all three?
11 — Coming Up Next
Tomorrow — Day 19: NLP Foundations. We leave images behind and enter the world of text. How do you turn words into numbers a model can reason over? Tokenisation, bag-of-words, TF-IDF, and the surprising power of simple text representations — the foundations that every modern language model, from BERT to GPT-4, is built on top of.
12 — Go Deeper
▶ YOLO Object Detection — Computerphile
YouTube · 15 min · Clear visual explanation of grid-based detection, anchors, and NMS.
🔗 Ultralytics YOLOv8 Docs
The definitive reference for training, evaluating, and deploying YOLO models. Includes detection, segmentation, pose, and classification tasks.
📄 U-Net: Convolutional Networks for Biomedical Image Segmentation — Ronneberger et al. (2015)
MICCAI 2015 · One of the most-cited papers in medical AI. Sections 2–3 explain the skip connection insight clearly. Essential reading.
📄 Segment Anything — Kirillov et al. (2023)
Meta AI · The SAM paper. Section 3 explains the data engine (how 1B masks were generated semi-automatically) — a masterclass in scaling data collection.
- Three levels of spatial understanding: classification (global label) → detection (label + bounding box) → segmentation (label per pixel). Each requires more annotation effort and produces richer spatial output.
- Two-stage detectors (Faster R-CNN) propose regions first, then classify — accurate but slow. One-stage detectors (YOLO) predict everything in one pass — fast enough for real-time.
- YOLO's key insight: frame detection as regression. Divide image into a grid; each cell directly predicts boxes and class labels. No region proposals. One forward pass, all detections simultaneously.
- Multi-scale detection (FPN neck) lets YOLO detect small objects in fine grids (80×80) and large objects in coarse grids (20×20) in the same pass. This is what handles far-away and close-up objects together.
- Non-Maximum Suppression (NMS) removes duplicate detections: keep the highest-confidence box, suppress all overlapping boxes (IoU above threshold) for the same class. Without NMS, every object would be reported 10–20 times.
- IoU = intersection area / union area. Used in loss, confidence thresholding, NMS, and evaluation means a detection is "correct" if IoU with ground truth > 0.5).
- U-Net's innovation = skip connections between encoder and decoder layers. Semantic understanding from the bottleneck + spatial precision from skip connections = sharp segmentation boundaries. Original 2015, still state of art in medical imaging.
- Dice score is the right metric for medical segmentation — it's not fooled by background dominance. Pixel accuracy can reach 99.9% by predicting only background. Dice = 0 in the same case.
- U-Net is literally inside Stable Diffusion. The denoising network is a U-Net. The same encoder-decoder-with-skip design works for both segmentation and image generation because both need global semantics + local spatial detail simultaneously.
- SAM (2023) generalised segmentation into a foundation model: segment any object from a point, box, or text prompt — zero-shot. The vision equivalent of GPT, bridging the gap between computer vision and language models.
- What is the difference between semantic and instance segmentation?
Semantic segmentation labels every pixel with a class but treats all instances identically — two people are both "person." Instance segmentation assigns a unique mask to each individual object — person #1 on the left, person #2 on the right. Instance segmentation is harder (requires knowing how many objects there are and their individual boundaries) and produces more useful output for tasks like cell counting, surgical tool tracking, or crowd analysis. - What is IoU and why does it appear in so many places in detection pipelines?
IoU (Intersection over Union) = area of overlap / area of union between two regions. It appears in: (1) the training loss — CIoU penalises imprecise predicted boxes; (2) confidence thresholding — does a predicted box sufficiently overlap a real object?; (3) Non-Maximum Suppression — are two boxes covering the same object (high IoU) or different ones?; (4) evaluation — a detection is "correct" if IoU with ground truth exceeds 0.5. It's the universal measure of spatial overlap quality, which is exactly what detection accuracy depends on. - Why is YOLO called "You Only Look Once," and what does that mean architecturally?
Unlike two-stage detectors that process the image twice (once for region proposals, once for classification), YOLO processes the image in a single forward pass and outputs all bounding boxes and class labels simultaneously. "One look" = one inference = all detections at once. This single-pass design is why YOLO achieves real-time speeds — 45 FPS for the original YOLO in 2015, 160+ FPS for YOLOv8n today — that two-stage detectors cannot match. - Why does U-Net use skip connections between encoder and decoder?
Encoder downsampling progressively loses spatial detail. The decoder reconstructing full-resolution predictions needs both the semantic understanding from the deep bottleneck (what the object is) and the precise spatial information from earlier encoder layers (where each boundary pixel falls). Skip connections carry high-resolution encoder feature maps directly to matching decoder levels, enabling sharp segmentation boundaries that upsampling alone cannot produce. - When would you use Dice score instead of accuracy for segmentation evaluation?
Whenever the class distribution is heavily imbalanced — which is almost always the case in medical imaging. A model predicting "background" for every pixel achieves 99.9% accuracy when the lesion covers <0.1% of the image. Dice = 2|A∩B|/(|A|+|B|) is 0 in this case, correctly identifying complete failure. Use Dice (or IoU) whenever the clinically or semantically important region is small relative to the total image area.
📄 U-Net: Convolutional Networks for Biomedical Image Segmentation — Ronneberger et al. (2015)
MICCAI 2015 · The U-Net paper. Sections 2–3 explain the architecture and skip connection rationale clearly. One of the most-cited papers in medical AI and a prerequisite for understanding diffusion models.
📄 Segment Anything — Kirillov et al., Meta AI (2023)
arXiv 2023 · The SAM paper. Section 3 (data engine) is particularly instructive on how to scale annotation collection semi-automatically — a broadly applicable technique beyond segmentation.
📄 You Only Look Once: Unified, Real-Time Object Detection — Redmon et al. (2015)
CVPR 2015 · The original YOLO paper. Unusually readable. Section 2 explains the grid-based regression reformulation that changed real-time detection forever.