Dropout
Dropout randomly zeros a fraction of activations during training, forcing the network to learn redundant representations. At inference, all units are active and outputs are scaled to compensate, acting as a regularizer.
What is Dropout?
Dropout randomly zeros a fraction of activations during training, forcing the network to learn redundant representations. At inference, all units are active and outputs are scaled to compensate, acting as a regularizer.
Dropout randomly zeros a fraction of activations during training, forcing the network to learn redundant representations. At inference, all units are active and outputs are scaled to compensate, acting as a regularizer.
Where is it used?
Applied to attention weights, FFN outputs, and residual connections in the original transformer and GPT-2 (typically 0.1-0.2 rate). Modern large models like LLaMA and GPT-3 often reduce or remove dropout, relying on data scale and other regularization instead.
How to build it
Add `nn.Dropout(p=0.1)` after attention output and FFN output in PyTorch. Call `model.eval()` before inference to disable dropout automatically; `model.train()` re-enables it.