Text Generation
Text generation uses the trained model to produce new text autoregressively. Starting from a prompt (or empty), the model predicts one token at a time, appends it, and repeats until a length limit or stop condition is met.
What is Text Generation?
Text generation uses the trained model to produce new text autoregressively. Starting from a prompt (or empty), the model predicts one token at a time, appends it, and repeats until a length limit or stop condition is met.
Text generation uses the trained model to produce new text autoregressively. Starting from a prompt (or empty), the model predicts one token at a time, appends it, and repeats until a length limit or stop condition is met.
Where is it used?
After training nanoGPT on Shakespeare, generation produces Shakespeare-like text. After training on TinyStories, it generates simple stories. This is the same mechanism used by GPT-4 and Claude, just at a smaller scale and with less coherent output.
How to build it
`def generate(model, ids, max_new, temperature=1.0): for _ in range(max_new): logits = model(ids)[:, -1] / temperature; probs = F.softmax(logits, dim=-1); next = torch.multinomial(probs, 1); ids = torch.cat([ids, next], dim=1); return ids`. Decode with `tokenizer.decode(ids[0])`.