Softmax
Softmax converts a vector of arbitrary real numbers into a probability distribution that sums to 1, exponentiating each element and normalizing. It turns attention scores into attention weights.
What is Softmax?
Softmax converts a vector of arbitrary real numbers into a probability distribution that sums to 1, exponentiating each element and normalizing. It turns attention scores into attention weights.
Softmax converts a vector of arbitrary real numbers into a probability distribution that sums to 1, exponentiating each element and normalizing. It turns attention scores into attention weights.
Where is it used?
Softmax is applied to the last dimension of attention scores in every transformer; it is also the final operation in LLM output heads that produces token probabilities.
How to build it
Use `F.softmax(logits, dim=-1)`, verify `.sum()` equals 1, and experiment with temperature by dividing logits by a scalar before softmax to sharpen or flatten the distribution.
Code
A practical example: