Generated Tokens
Generated tokens are the output tokens produced by the model after the prompt. They are created one at a time (or in speculative batches) and appended to the sequence. The count of generated tokens determines the response length and output cost.
What is Generated Tokens?
Generated tokens are the output tokens produced by the model after the prompt. They are created one at a time (or in speculative batches) and appended to the sequence. The count of generated tokens determines the response length and output cost.
Generated tokens are the output tokens produced by the model after the prompt. They are created one at a time (or in speculative batches) and appended to the sequence. The count of generated tokens determines the response length and output cost.
Where is it used?
LLM APIs bill based on generated tokens (OpenAI charges per output token at a higher rate than input). vLLM and TGI track generated token counts for rate limiting and billing. The generation loop runs until the token budget or a stop condition is met.
How to build it
Track: `generated = ids[:, prompt_len:]` after the generation loop. Count: `n_generated = generated.shape[1]`. Decode: `output_text = tokenizer.decode(generated[0], skip_special_tokens=True)`.