Concatenation
Concatenation combines the outputs of all attention heads along the feature dimension, forming a single tensor of width `num_heads * d_k = d_model` that is ready for the output projection.
What is Concatenation?
Concatenation combines the outputs of all attention heads along the feature dimension, forming a single tensor of width `num_heads * d_k = d_model` that is ready for the output projection.
Concatenation combines the outputs of all attention heads along the feature dimension, forming a single tensor of width `num_heads * d_k = d_model` that is ready for the output projection.
Where is it used?
After per-head attention in every transformer layer, the heads are concatenated before being multiplied by W_O; this recombination lets the model mix information captured by different heads.
How to build it
Reshape each head output to `(batch, seq, d_k)`, `torch.cat(heads, dim=-1)` to get `(batch, seq, d_model)`, and verify the dimension matches the input width for the residual connection.