Model Width
Model width is the hidden dimension (d_model) of the transformer, the size of the vector representing each token. Wider models can store more information per token but increase parameter count quadratically in attention and FFN layers.
What is Model Width?
Model width is the hidden dimension (d_model) of the transformer, the size of the vector representing each token. Wider models can store more information per token but increase parameter count quadratically in attention and FFN layers.
Model width is the hidden dimension (d_model) of the transformer, the size of the vector representing each token. Wider models can store more information per token but increase parameter count quadratically in attention and FFN layers.
Where is it used?
GPT-2 small uses d_model=768, GPT-3 175B uses 12288, LLaMA-2 7B uses 4096, and LLaMA-2 70B uses 8192. Width is the primary lever for model capacity and is tied to head dimension (`d_model / n_heads`).
How to build it
Set `d_model` in the config. It determines embedding size, attention projection size, and FFN intermediate size (`d_ff = 4 * d_model`). Choose `head_dim = d_model / n_heads` (typically 64 or 128). Embedding: `nn.Embedding(vocab, d_model)`.