Transformer
The transformer is the core architecture of the small LLM: a stack of decoder-only blocks, each with causal self-attention and a feed-forward network, connected by residual connections and layer normalization.
What is Transformer?
The transformer is the core architecture of the small LLM: a stack of decoder-only blocks, each with causal self-attention and a feed-forward network, connected by residual connections and layer normalization.
The transformer is the core architecture of the small LLM: a stack of decoder-only blocks, each with causal self-attention and a feed-forward network, connected by residual connections and layer normalization.
Where is it used?
A small transformer with 6-12 layers, 6-12 heads, and d_model=384-768 can learn basic language patterns from a book dataset. Karpathy's nanoGPT uses 6 layers, 6 heads, 384 dims for Shakespeare. This same architecture scales to GPT-3 175B with more layers and width.
How to build it
Define `TransformerBlock` with `LayerNorm → CausalSelfAttention → residual → LayerNorm → MLP → residual`. Stack N blocks in `GPT`. Add final `LayerNorm` and `Linear(d_model, vocab_size)` LM head. Initialize weights with `nn.init.normal_` (std=0.02) and scale residuals by `1/sqrt(2*n_layer)`.