Monitoring
Monitoring tracks inference latency, throughput, GPU utilisation, error rates, and queue depth in real time to detect degradation and trigger autoscaling or alerts.
What is Monitoring?
Monitoring tracks inference latency, throughput, GPU utilisation, error rates, and queue depth in real time to detect degradation and trigger autoscaling or alerts.
Monitoring tracks inference latency, throughput, GPU utilisation, error rates, and queue depth in real time to detect degradation and trigger autoscaling or alerts.
Where is it used?
Prometheus + Grafana, NVIDIA DCGM, and Datadog monitor vLLM/TGI metrics; production teams alert on TTFT p99 and GPU memory pressure.
How to build it
Scrape vLLM's `/metrics` endpoint with Prometheus, build a Grafana dashboard for `vllm:num_requests_waiting`, `vllm:time_to_first_token_seconds`, and GPU `DCGM_FI_DEV_GPU_UTIL`.