Residual Connection
A residual (skip) connection adds a layer's input back to its output: `x = x + Sublayer(x)`. This creates a gradient highway that lets signals flow directly to earlier layers, preventing degradation in deep networks.
What is Residual Connection?
A residual (skip) connection adds a layer's input back to its output: `x = x + Sublayer(x)`. This creates a gradient highway that lets signals flow directly to earlier layers, preventing degradation in deep networks.
A residual (skip) connection adds a layer's input back to its output: `x = x + Sublayer(x)`. This creates a gradient highway that lets signals flow directly to earlier layers, preventing degradation in deep networks.
Where is it used?
Present in every transformer block of GPT-2, BERT, LLaMA, and Mistral around the attention and FFN sublayers. Modern models like LLaMA and GPT-2 use pre-norm residuals (`x = x + Sublayer(LayerNorm(x))`) for better training stability than the original post-norm design.
How to build it
Simply add the input to the sublayer output: `x = x + self_attn(x)` and `x = x + ffn(x)` in PyTorch. With pre-norm: `x = x + self_attn(self.ln1(x))` then `x = x + ffn(self.ln2(x))`.