Bias
Bias is an additive learnable constant in a neuron that shifts the activation independently of the input, allowing the model to fit data that does not pass through the origin.
What is Bias?
Bias is an additive learnable constant in a neuron that shifts the activation independently of the input, allowing the model to fit data that does not pass through the origin.
Bias is an additive learnable constant in a neuron that shifts the activation independently of the input, allowing the model to fit data that does not pass through the origin.
Where is it used?
Every `nn.Linear` in a transformer has a bias (except in some architectures like Llama which omit bias in attention); biases give layers the flexibility to threshold activations.
How to build it
Create `nn.Linear(3,1)` and print `layer.bias`; train a model with and without bias (`bias=False`) on offset data to see why bias matters for fit.