Continuous Batching
Continuous batching (in-flight batching) dynamically inserts and evicts requests from a running batch at every decode step, keeping the GPU saturated even as requests finish at different times.
What is Continuous Batching?
Continuous batching (in-flight batching) dynamically inserts and evicts requests from a running batch at every decode step, keeping the GPU saturated even as requests finish at different times.
Continuous batching (in-flight batching) dynamically inserts and evicts requests from a running batch at every decode step, keeping the GPU saturated even as requests finish at different times.
Where is it used?
vLLM's `ContinuousBatchingExecutor`, TGI, and SGLang use continuous batching to achieve 10-20x higher throughput than static batching for variable-length chat workloads.
How to build it
Serve Llama-3-8B with `python -m vllm.entrypoints.openai.api_server --model meta-llama/Meta-Llama-3-8B-Instruct --max-num-batched-tokens 4096` and load-test with `locust` to observe throughput scaling.