Transfer learning is the practice of reusing the knowledge a model gained on one task to solve a second, related one. It is like a pianist who learns the organ far faster than a beginner: the musical foundations carry over.
Why it works
A deep neural network learns hierarchical representations. Early layers capture generic patterns (edges, textures, shapes), while later layers specialize toward the target task. Because low-level patterns are shared across many problems, we can keep those layers and retrain only the specialized part. This is the idea of a pre-trained model (trained on a large dataset such as ImageNet) that we then adapt.
Two common strategies
| Strategy | Frozen layers | Data needed | Cost |
|---|---|---|---|
| Feature extraction | All but the head | Little | Low |
| Fine-tuning | None (or partial) | Moderate | Higher |
In feature extraction we freeze the network backbone ($\theta_{\text{base}}$ fixed) and train only the new classification head. In fine-tuning, we gently readjust all weights with a small learning rate:
$$\theta^* = \arg\min_{\theta} \; \mathcal{L}{\text{target}}(\theta), \quad \theta$$}} = \theta_{\text{source}
Why it matters
- Drastically reduces the need for labeled data.
- Speeds up training and lowers compute costs.
- Makes AI accessible to small teams without data centers.
This is the principle behind most of today's large models (vision, language), where a foundation model is adapted to a specific use case.
Learn once, reuse everywhere: transfer turns general knowledge into targeted skill.