Picture a reverse coloring book: instead of filling in pre-drawn shapes, the machine must draw the outlines itself, deciding pixel by pixel which object each one belongs to. That is exactly what image segmentation does: assign a label to every pixel so the scene is carved into coherent regions (road, pedestrian, tumor, sky…).
Three main families
There are three levels of granularity:
| Type | Question asked | Example |
|---|---|---|
| Semantic | What class is this pixel? | "car", "road" |
| Instance | Which specific object? | car #1 ≠ car #2 |
| Panoptic | Class + instance combined | every pixel labeled |
Semantic segmentation groups all pixels of the same class together, while instance segmentation tells individual objects apart.
How it works
Modern architectures rely on encoder–decoder networks such as U-Net or Fully Convolutional Networks (FCN). The encoder compresses the image into abstract features; the decoder rebuilds a full-resolution segmentation map. More recently, Meta's SAM (Segment Anything Model) popularized general-purpose, promptable segmentation.
To measure quality, we use the Jaccard index (Intersection over Union):
$$\text{IoU} = \frac{|A \cap B|}{|A \cup B|}$$
where $A$ is the prediction and $B$ the ground truth. An IoU near 1 means near-perfect overlap.
Where it is used
- Self-driving cars: isolate pedestrians, lanes, obstacles
- Medical imaging: outline organs and lesions
- Remote sensing: map crops and forests
- Photo editing: cut out a subject automatically
Where classification tells you what is in an image, segmentation tells you where it is — down to the pixel.