Activation Functions
An activation function introduces non-linearity into a neuron's output, enabling the network to approximate complex functions. Common choices are ReLU, GELU, and SiLU (SwiGLU).
What is Activation Functions?
An activation function introduces non-linearity into a neuron's output, enabling the network to approximate complex functions. Common choices are ReLU, GELU, and SiLU (SwiGLU).
An activation function introduces non-linearity into a neuron's output, enabling the network to approximate complex functions. Common choices are ReLU, GELU, and SiLU (SwiGLU).
Where is it used?
Transformers use GELU (GPT-2) or SiLU within SwiGLU (Llama, PaLM) in their MLP layers; without nonlinearity, stacking layers would collapse into a single linear map.
How to build it
Plot `F.relu`, `F.gelu`, and `F.silu` over `torch.linspace(-5,5,100)` with matplotlib, and build a 2-layer net with and without activation to show nonlinearity's effect.
Code
A practical example: