Batch Inference
Batch inference processes multiple prompts together in a single forward pass, improving GPU utilisation and throughput by parallelising the matmuls across the batch dimension.
What is Batch Inference?
Batch inference processes multiple prompts together in a single forward pass, improving GPU utilisation and throughput by parallelising the matmuls across the batch dimension.
Batch inference processes multiple prompts together in a single forward pass, improving GPU utilisation and throughput by parallelising the matmuls across the batch dimension.
Where is it used?
Offline eval pipelines and vLLM/TensorRT-LLM serving use dynamic batching; HF `generate()` accepts padded batches via `tokenizer(prompts, padding=True, return_tensors="pt")`.
How to build it
Tokenise a list of prompts with padding, run `model.generate(input_ids, attention_mask=attention_mask)` once for the batch, and compare wall time vs looping one-by-one.