Model Depth
Model depth is the number of transformer layers (blocks) stacked in the model. More layers allow the model to learn more abstract, hierarchical representations but increase training time, memory, and risk of gradient degradation.
What is Model Depth?
Model depth is the number of transformer layers (blocks) stacked in the model. More layers allow the model to learn more abstract, hierarchical representations but increase training time, memory, and risk of gradient degradation.
Model depth is the number of transformer layers (blocks) stacked in the model. More layers allow the model to learn more abstract, hierarchical representations but increase training time, memory, and risk of gradient degradation.
Where is it used?
GPT-2 small has 12 layers, GPT-2 XL has 48, LLaMA-2 7B has 32, and LLaMA-2 70B has 80. Depth and width are co-tuned: deeper-narrower models (GPT-2 XL) vs shallower-wider ones. Scaling laws suggest depth matters less than total params.
How to build it
Set `n_layers` as a model config hyperparameter and stack blocks: `self.blocks = nn.ModuleList([TransformerBlock(config) for _ in range(n_layers)])`. In the forward: `for block in self.blocks: x = block(x)`.