Output Projection
The output projection is a learned linear map W_O applied to the concatenated multi-head output, mixing information across heads and producing the final attention layer output added to the residual stream.
What is Output Projection?
The output projection is a learned linear map W_O applied to the concatenated multi-head output, mixing information across heads and producing the final attention layer output added to the residual stream.
The output projection is a learned linear map W_O applied to the concatenated multi-head output, mixing information across heads and producing the final attention layer output added to the residual stream.
Where is it used?
Every `nn.MultiheadAttention` ends with `W_O: nn.Linear(d_model, d_model)`; in Llama and GPT, this projection's output is added to the input (residual) and then passed through LayerNorm.
How to build it
Apply `W_O = nn.Linear(d_model, d_model)` to the concatenated tensor, add the result to the original input `x + attn_out`, and verify the output shape matches for the next layer.