Request Handling
Request handling is the process of receiving an inference request, tokenising the prompt, scheduling it into a batch, running generation, detokenising, and returning the response.
What is Request Handling?
Request handling is the process of receiving an inference request, tokenising the prompt, scheduling it into a batch, running generation, detokenising, and returning the response.
Request handling is the process of receiving an inference request, tokenising the prompt, scheduling it into a batch, running generation, detokenising, and returning the response.
Where is it used?
vLLM's `LLMEngine` and TGI's internal scheduler handle requests; they manage waiting queues, continuous batching, and cancellation for streaming clients.
How to build it
In a FastAPI handler, call `llm.generate(prompt, sampling_params)`, iterate the output generator, and yield `data: {json.dumps(chunk)}\n\n` chunks for SSE streaming.