LLM Inference
LLM inference is the process of running a trained model to generate text from a prompt, comprising a prefill phase (processing the prompt) and a decode phase (generating tokens autoregressively).
What is LLM Inference?
LLM inference is the process of running a trained model to generate text from a prompt, comprising a prefill phase (processing the prompt) and a decode phase (generating tokens autoregressively).
LLM inference is the process of running a trained model to generate text from a prompt, comprising a prefill phase (processing the prompt) and a decode phase (generating tokens autoregressively).
Where is it used?
Every chatbot API call (OpenAI, Anthropic, vLLM-served Llama) performs inference; inference is compute- and memory-bound, dominated by attention and matmul operations.
How to build it
Load a model with `AutoModelForCausalLM.from_pretrained`, call `model.generate(input_ids, max_new_tokens=100)`, and decode with `tokenizer.decode(output[0])` to see generated text.