Self-Attention
Self-attention is attention where the queries, keys, and values all come from the same input sequence, allowing each token to weigh every other token's relevance to itself. It captures intra-sequence relationships.
What is Self-Attention?
Self-attention is attention where the queries, keys, and values all come from the same input sequence, allowing each token to weigh every other token's relevance to itself. It captures intra-sequence relationships.
Self-attention is attention where the queries, keys, and values all come from the same input sequence, allowing each token to weigh every other token's relevance to itself. It captures intra-sequence relationships.
Where is it used?
GPT, Llama, and BERT are built entirely from self-attention layers; each token's updated representation is a weighted mixture of all tokens, weighted by learned similarity.
How to build it
Use `nn.MultiheadAttention(embed_dim, num_heads, batch_first=True)` with `query=key=value=x`, run a forward pass, and inspect the output and attention weights.
Code
A practical example:
Further reading
Want to go deeper? These resources cover Self-Attention in more detail: