Embedding Parameters
Embedding parameters map discrete token IDs to dense vectors. The token embedding matrix has shape `(vocab_size, d_model)`, and positional embeddings (learned or rotary) encode position. These are the first parameters input data touches.
What is Embedding Parameters?
Embedding parameters map discrete token IDs to dense vectors. The token embedding matrix has shape `(vocab_size, d_model)`, and positional embeddings (learned or rotary) encode position. These are the first parameters input data touches.
Embedding parameters map discrete token IDs to dense vectors. The token embedding matrix has shape `(vocab_size, d_model)`, and positional embeddings (learned or rotary) encode position. These are the first parameters input data touches.
Where is it used?
GPT-2 uses learned positional embeddings; LLaMA and Mistral use rotary positional embeddings (RoPE) which add no parameters. The token embedding is often the largest single matrix: GPT-2's is 50257 x 768 ≈ 38M params.
How to build it
Token: `nn.Embedding(vocab_size, d_model)`. Positional (learned): `nn.Embedding(max_seq_len, d_model)`. For RoPE, precompute rotation frequencies and apply them to Q/K vectors without learnable params. Tie LM head: `lm_head.weight = embed.weight`.