Embedding Matrix
The embedding matrix is a learnable weight matrix of shape (vocab_size, d_model) where row i is the embedding vector for token ID i. It is essentially a lookup table implemented as a matrix.
What is Embedding Matrix?
The embedding matrix is a learnable weight matrix of shape (vocab_size, d_model) where row i is the embedding vector for token ID i. It is essentially a lookup table implemented as a matrix.
The embedding matrix is a learnable weight matrix of shape (vocab_size, d_model) where row i is the embedding vector for token ID i. It is essentially a lookup table implemented as a matrix.
Where is it used?
The embedding matrix is shared in many LLMs (weight tying) with the output projection so that the same matrix maps tokens to vectors and hidden states back to vocab logits.
How to build it
Create `emb = nn.Embedding(10000, 512)`, access a single token's vector with `emb.weight[42]`, and multiply a one-hot vector by `emb.weight.T` to prove it equals `emb(token_id)`.
Code
A practical example: