Embedding Dimension
The embedding dimension (d_model) is the size of the vector used to represent each token. Higher dimensions capture more information but increase compute and memory quadratically in attention.
What is Embedding Dimension?
The embedding dimension (d_model) is the size of the vector used to represent each token. Higher dimensions capture more information but increase compute and memory quadratically in attention.
The embedding dimension (d_model) is the size of the vector used to represent each token. Higher dimensions capture more information but increase compute and memory quadratically in attention.
Where is it used?
GPT-2 small uses 768; Llama-2 7B uses 4096; Llama-3 70B uses 8192; choosing d_model balances capacity against training cost and latency.
How to build it
Compare `nn.Embedding(32000, 512)` vs `nn.Embedding(32000, 4096)` parameter counts, and measure attention layer FLOPs scaling with d_model to see the quadratic cost.