Picture a giant mixing console with millions, even billions, of knobs. Each one finely tunes how information flows through the model. Those knobs are the parameters: the internal numerical values a neural network adjusts during training to turn an input (text, an image) into a meaningful output.
Weights and biases: the model's memory
A parameter is, concretely, a number. There are two families:
- weights, which scale the importance of each connection between neurons;
- biases, which shift the result to adjust the activation threshold.
For a single neuron, the output is computed as:
$$y = \sigma\left(\sum_{i=1}^{n} w_i x_i + b\right)$$
where the $w_i$ are the weights, $b$ the bias, and $\sigma$ an activation function. Learning is precisely the search for the right values of these $w$ and $b$, via gradient descent.
How many parameters?
The parameter count has become a measure of a model's "size."
| Model | Order of magnitude |
|---|---|
| Small classic model | Thousands to millions |
| GPT-2 | ~1.5 billion |
| Recent large language models | Tens to hundreds of billions |
Be careful: more parameters does not always mean "better." An oversized model can overfit (memorize instead of generalize) and is hugely expensive in compute and energy.
Not to be confused
Parameters (learned by the machine) differ from hyperparameters (set by humans: learning rate, number of layers). The former emerge from training; the latter steer it.
Parameters are a model's frozen knowledge: everything it has "learned" lives inside those few billion numbers.