Embeddings
The embedding layer converts token IDs into dense vectors of dimension `d_model`, which is the model's internal representation size. These vectors are learnable parameters that the model adjusts to encode semantic and positional information about each token.
What is Embeddings?
The embedding layer converts token IDs into dense vectors of dimension `d_model`, which is the model's internal representation size. These vectors are learnable parameters that the model adjusts to encode semantic and positional information about each token.
The embedding layer converts token IDs into dense vectors of dimension `d_model`, which is the model's internal representation size. These vectors are learnable parameters that the model adjusts to encode semantic and positional information about each token.
Where is it used?
Every transformer (GPT-2, LLaMA, BERT) starts with an embedding layer. GPT-2's token embedding is 50257 x 768. In small educational models, embeddings might be just 65 x 384 (Shakespeare char-level). The embedding matrix is often tied to the output LM head.
How to build it
`self.token_embedding = nn.Embedding(vocab_size, d_model)`. Optionally add positional: `self.pos_embedding = nn.Embedding(block_size, d_model)`. Forward: `x = self.token_embedding(ids) + self.pos_embedding(torch.arange(T))`. Or use RoPE (no params).