Picture an assistant that can both look at a photo and talk to you about it: that is exactly what a Vision-Language Model (VLM) is. It fuses the visual perception of a vision model with the understanding and text generation of a large language model, inside a single artificial brain.
How it works
A VLM rests on three building blocks. An image encoder (often a Vision Transformer) splits the image into patches and turns them into vectors. A projector aligns these vectors into the same space as words. Finally, the language model reads this mixed sequence (image + text) and generates an answer.
The key idea is multimodal alignment: pixels and words must live in a shared space. The famous CLIP model (OpenAI, 2021) popularized this by learning to pull an image close to its caption:
$$\text{sim}(I, T) = \frac{f(I) \cdot g(T)}{\lVert f(I)\rVert \, \lVert g(T)\rVert}$$
where $f(I)$ encodes the image and $g(T)$ the text; a high score means they "talk about" the same thing.
What it is for
| Task | Concrete example |
|---|---|
| Captioning | Automatically describe a photo |
| VQA | Answer "how many people are in this image?" |
| OCR + reasoning | Read a receipt and compute the total |
| Assistance | Guide a visually impaired user, analyze a medical chart |
Limits
VLMs can hallucinate objects that are not there, misread fine print, or inherit biases present in their training data. Their answer is a probability, not a guaranteed truth.
A VLM does more than see: it links the visible to language, paving the way for AI that understands the world the way we describe it.