Parameters versus hyperparameters
A parameter is learned by the optimiser. A hyperparameter is chosen by the person running the training: learning rate, batch size, number of layers, embedding width, dropout rate. You can freeze a parameter, but you cannot learn a hyperparameter from gradients alone.
This distinction matters when someone fine-tunes a model. A LoRA adapter does not change the base parameters; it adds a small pair of low-rank matrices whose parameters are trained. That is why an adapter can be a few megabytes while the base model is several gigabytes.
| Kind | Example | Who sets it? |
|---|---|---|
| Parameter | A weight in an attention projection | Learned from data |
| Parameter | A token embedding vector | Learned from data |
| Hyperparameter | Learning rate 3e-4 | Chosen by the engineer |
| Hyperparameter | 32 layers, 4096-wide hidden state | Chosen by the architect |
Counting parameters in a layer
A fully connected layer with n_in inputs and n_out outputs needs one weight per input–output pair, plus one bias per output:
Take a small classifier that takes 784 pixel values, has one hidden layer of 128 ReLU units, and outputs 10 class scores. The first layer has 784 × 128 + 128 = 100,480 parameters. The second has 128 × 10 + 10 = 1,290. The total is 101,770 — small enough to train on a laptop.
Transformer blocks are dominated by four attention projections (about 4d²) and a feed-forward network usually four times wider (about 8d²), so roughly 12d² per block before embeddings and normalisation. That is why doubling the hidden width roughly quadruples the parameter count.
Why the count is used as shorthand
Parameter count predicts memory. Storing a 7B-parameter model at 16-bit precision needs about 14 GB (13 GiB) before any cache or activations — the arithmetic is parameter count times bytes per value. At 4-bit quantization the weights fall to roughly 3.5 GB, which is why quantized models run on smaller GPUs.
- Inference memory ≈ weights + KV cache + temporary buffers.
- Training memory adds gradients, optimiser state and cached activations — often several times the weight size.
- Fine-tuning with LoRA trains a small fraction of the parameters, so optimiser memory shrinks dramatically.
- Mixture-of-experts models quote a total parameter count but activate only a subset per token, so total and active parameters differ.
More parameters is not automatically better
A larger model has more capacity, but capacity only helps when there is enough high-quality data and compute to use it. Scaling studies consistently show that model size, data size and compute need to grow together; an oversized model trained on too little data underperforms a smaller, better-trained one.
The practical costs are real: bigger models are slower, more expensive to serve, harder to fit on a device and often harder to evaluate. Distillation, quantization and smaller specialised models exist because “largest available” is rarely the right engineering answer.
To turn a parameter count into a hardware plan, use the VRAM calculator; to see what an adapter costs to train, use the fine-tuning memory calculator. Stage 9 covers LoRA and Stage 2 shows where these weights live in a neuron.