Scaled Dot-Product Attention
Scaled dot-product attention computes `softmax(QK^T / sqrt(d_k)) V`, scaling by sqrt(d_k) to keep gradients stable as d_k grows. It is the canonical attention formula from the original Transformer paper.
What is Scaled Dot-Product Attention?
Scaled dot-product attention computes `softmax(QK^T / sqrt(d_k)) V`, scaling by sqrt(d_k) to keep gradients stable as d_k grows. It is the canonical attention formula from the original Transformer paper.
Scaled dot-product attention computes `softmax(QK^T / sqrt(d_k)) V`, scaling by sqrt(d_k) to keep gradients stable as d_k grows. It is the canonical attention formula from the original Transformer paper.
Where is it used?
This exact formula is the core operation in every GPT, Llama, and BERT attention layer; `F.scaled_dot_product_attention` in PyTorch 2.0 fuses it into a fast Flash Attention kernel.
How to build it
Call `F.scaled_dot_product_attention(Q, K, V, is_causal=True)` and compare with a manual implementation to verify identical outputs and faster runtime on GPU.