GPT Architecture
GPT (Generative Pre-trained Transformer) is a decoder-only transformer pretrained on next-token prediction over large text corpora. It uses learned positional embeddings, pre-norm LayerNorm, GELU FFN, and is fine-tuned or prompted for downstream tasks.
What is GPT Architecture?
GPT (Generative Pre-trained Transformer) is a decoder-only transformer pretrained on next-token prediction over large text corpora. It uses learned positional embeddings, pre-norm LayerNorm, GELU FFN, and is fine-tuned or prompted for downstream tasks.
GPT (Generative Pre-trained Transformer) is a decoder-only transformer pretrained on next-token prediction over large text corpora. It uses learned positional embeddings, pre-norm LayerNorm, GELU FFN, and is fine-tuned or prompted for downstream tasks.
Where is it used?
The GPT family ranges from GPT-1 (117M) to GPT-4 (trillion-scale). OpenAI's GPT-2 and GPT-3 are the canonical open-weight references. LLaMA, Mistral, and Falcon follow the same decoder-only GPT template with variations like RMSNorm and RoPE.
How to build it
Implement a `GPTModel` with `nn.Embedding` for tokens and positions, a stack of decoder blocks, final LayerNorm, and an LM head tied to the embedding weights. Karpathy's nanoGPT is a minimal reference (~300 lines).
Code
A practical example:
Further reading
Want to go deeper? These resources cover GPT Architecture in more detail: