Inference Experiments
Inference experiments vary serving parameters (quantisation, batch size, speculative decoding, chunk size) to study their effect on latency, throughput, and quality.
What is Inference Experiments?
Inference experiments vary serving parameters (quantisation, batch size, speculative decoding, chunk size) to study their effect on latency, throughput, and quality.
Inference experiments vary serving parameters (quantisation, batch size, speculative decoding, chunk size) to study their effect on latency, throughput, and quality.
Where is it used?
vLLM `benchmark_throughput.py` and TensorRT-LLM benchmark suites run inference experiments; the FlashAttention and speculative-decoding papers report these trade-off curves.
How to build it
Sweep `--max-num-batched-tokens [1024, 4096, 16384]` and `--quantization [none, awq, fp8]` in vLLM, log TTFT and tokens/s with `benchmark_serving.py`, and plot the Pareto frontier.