Parameter Count
Parameter count is the total number of individual learnable values in a model. It is computed by summing the sizes of all weight matrices and vectors, and serves as the primary metric for comparing model scale.
What is Parameter Count?
Parameter count is the total number of individual learnable values in a model. It is computed by summing the sizes of all weight matrices and vectors, and serves as the primary metric for comparing model scale.
Parameter count is the total number of individual learnable values in a model. It is computed by summing the sizes of all weight matrices and vectors, and serves as the primary metric for comparing model scale.
Where is it used?
Model cards report parameter counts: LLaMA-2 7B/13B/70B, Mistral 7B, GPT-3 175B. The count drives memory requirements (count x bytes per param), training compute (Chinchilla scaling laws), and inference latency.
How to build it
Compute analytically: for a decoder-only model, parameters ≈ `12 * n_layers * d_model²` (attention + FFN) plus embeddings `2 * vocab * d_model` (if untied). In code: `sum(p.numel() for p in model.parameters())`.
Code
A practical example: