Token Embeddings
Token embeddings are dense vectors learned by the model that represent each token's meaning, mapping discrete token IDs into a continuous space where similar words have similar vectors.
What is Token Embeddings?
Token embeddings are dense vectors learned by the model that represent each token's meaning, mapping discrete token IDs into a continuous space where similar words have similar vectors.
Token embeddings are dense vectors learned by the model that represent each token's meaning, mapping discrete token IDs into a continuous space where similar words have similar vectors.
Where is it used?
Every transformer's first layer is an embedding lookup: GPT-2 maps 50,257 tokens to 768-dim vectors; Llama 3 maps 128k tokens to 4096-dim vectors that flow into the first attention block.
How to build it
Use `nn.Embedding(vocab_size, d_model)`, pass token IDs `embed(tokens)`, and inspect the output shape `(batch, seq_len, d_model)`; visualize embeddings with t-SNE after training.
Code
A practical example:
Further reading
Want to go deeper? These resources cover Token Embeddings in more detail: