Compute Optimization
Compute optimisation for inference maximises GPU FLOPs utilisation via kernel fusion, FlashAttention, tensor cores (bf16/fp8), and operator-level compilation to reduce latency and cost.
What is Compute Optimization?
Compute optimisation for inference maximises GPU FLOPs utilisation via kernel fusion, FlashAttention, tensor cores (bf16/fp8), and operator-level compilation to reduce latency and cost.
Compute optimisation for inference maximises GPU FLOPs utilisation via kernel fusion, FlashAttention, tensor cores (bf16/fp8), and operator-level compilation to reduce latency and cost.
Where is it used?
TensorRT-LLM compiles Llama/Mistral into fused kernels; `torch.compile(model, mode="max-autotune")` and vLLM's custom CUDA kernels improve compute throughput on A100/H100.
How to build it
Wrap the model with `torch.compile(model, mode="max-autotune-no-cudagraphs")`, warm up with a dummy forward, and benchmark generation tokens/s against the eager baseline.