GPU Allocation
GPU allocation is the assignment of model replicas, pipeline stages, or KV-cache memory to specific GPUs to maximise utilisation while respecting VRAM and latency constraints.
What is GPU Allocation?
GPU allocation is the assignment of model replicas, pipeline stages, or KV-cache memory to specific GPUs to maximise utilisation while respecting VRAM and latency constraints.
GPU allocation is the assignment of model replicas, pipeline stages, or KV-cache memory to specific GPUs to maximise utilisation while respecting VRAM and latency constraints.
Where is it used?
vLLM `--tensor-parallel-size` and Ray Serve manage GPU allocation; Kubernetes device plugin and NVIDIA MIG partition H100s for multi-tenant LLM serving.
How to build it
Launch vLLM with `--tensor-parallel-size 4 --gpu-memory-utilization 0.9` across 4 GPUs, monitor `nvidia-smi` per-GPU VRAM, and adjust `--max-num-batched-tokens` to fit KV cache.