Feed-Forward Network
The feed-forward network (FFN) is a two-layer MLP applied to each token position independently after the attention sublayer. It expands the hidden dimension to a larger intermediate size, applies a non-linearity, and projects back, giving the model per-token nonlinear capacity.
What is Feed-Forward Network?
The feed-forward network (FFN) is a two-layer MLP applied to each token position independently after the attention sublayer. It expands the hidden dimension to a larger intermediate size, applies a non-linearity, and projects back, giving the model per-token nonlinear capacity.
The feed-forward network (FFN) is a two-layer MLP applied to each token position independently after the attention sublayer. It expands the hidden dimension to a larger intermediate size, applies a non-linearity, and projects back, giving the model per-token nonlinear capacity.
Where is it used?
Every transformer block in GPT, BERT, LLaMA, and T5 contains an FFN sublayer. GPT-style models typically use a 4x expansion ratio; LLaMA swaps the ReLU/GELU for SwiGLU, increasing the expansion to ~8/3x to keep parameter count constant.
How to build it
In PyTorch: `nn.Sequential(nn.Linear(d_model, d_ff), nn.GELU(), nn.Linear(d_ff, d_model))` where `d_ff = 4 * d_model`. For SwiGLU, implement a gated variant: `Linear(d_model, d_ff) * silu(Linear(d_model, d_ff))` then `Linear(d_ff, d_model)`.