Feed-Forward Parameters
Feed-forward parameters are the weights of the two (or three for gated) linear layers in the FFN sublayer. They typically account for ~2/3 of a transformer's total parameters because the intermediate dimension is 4x the model dimension.
What is Feed-Forward Parameters?
Feed-forward parameters are the weights of the two (or three for gated) linear layers in the FFN sublayer. They typically account for ~2/3 of a transformer's total parameters because the intermediate dimension is 4x the model dimension.
Feed-forward parameters are the weights of the two (or three for gated) linear layers in the FFN sublayer. They typically account for ~2/3 of a transformer's total parameters because the intermediate dimension is 4x the model dimension.
Where is it used?
In GPT-2 (d_model=768, d_ff=3072), the FFN has ~4.7M params per layer vs ~2.4M for attention. LLaMA's SwiGLU FFN uses three matrices (gate, up, down) with d_ff ≈ 2.67x d_model to keep param count constant relative to ReLU FFN.
How to build it
Standard FFN: `Linear(d_model, 4*d_model)` + `Linear(4*d_model, d_model)` = `8 * d_model²` params per layer. SwiGLU: three matrices `Linear(d, d_ff)` x2 + `Linear(d_ff, d)` = `3 * d * d_ff`. Count: `sum(p.numel() for p in ffn.parameters())`.