Word2Vec is a family of models introduced by a Google team in 2013, led by Tomas Mikolov, that learns to represent each word as a dense vector of real numbers. Its intuition rests on a famous linguistic principle: "you shall know a word by the company it keeps." Word2Vec therefore places words with similar meanings in neighboring regions of the vector space.
Two complementary architectures
Word2Vec relies on a simple neural network (a single hidden layer) trained in two variants:
| Architecture | Principle |
|---|---|
| CBOW (Continuous Bag-of-Words) | predicts the center word from its context |
| Skip-gram | predicts the context from the center word |
Skip-gram works better for rare words; CBOW is faster. To make training efficient on large corpora, techniques such as negative sampling and subsampling of frequent words are used.
The arithmetic of meaning
Word2Vec's most striking property is that semantic relations become vector operations. The canonical example:
$$\vec{king} - \vec{man} + \vec{woman} \approx \vec{queen}$$
The closeness between two words is measured by cosine similarity:
$$\text{sim}(\vec{u}, \vec{v}) = \frac{\vec{u} \cdot \vec{v}}{|\vec{u}|\,|\vec{v}|}$$
Legacy
- Popularized static word embeddings.
- Paved the way for GloVe, FastText, and later contextual models (BERT, GPT).
- Major limitation: one word = one vector, so it cannot handle polysemy ("bank" as a riverside vs. a financial institution).
Word2Vec turned language into geometry: understanding a word now means measuring a distance.