Token Generation
Token generation is the autoregressive emission of tokens one at a time, where each new token is sampled (greedy, top-p, top-k) from the model's output distribution conditioned on all prior tokens.
What is Token Generation?
Token generation is the autoregressive emission of tokens one at a time, where each new token is sampled (greedy, top-p, top-k) from the model's output distribution conditioned on all prior tokens.
Token generation is the autoregressive emission of tokens one at a time, where each new token is sampled (greedy, top-p, top-k) from the model's output distribution conditioned on all prior tokens.
Where is it used?
OpenAI `temperature`, `top_p`, and `max_tokens` parameters control generation; HF `generate(do_sample=True, top_k=50, top_p=0.9)` implements the same sampling strategies.
How to build it
Use `model.generate(input_ids, do_sample=True, temperature=0.7, top_p=0.9, max_new_tokens=100)` and inspect `generation_config` to see the sampling parameters in effect.