LibraryProgress

Generated Files

llm-course/home

Topics

MDComputer & Python Foundations7 filesMDMathematics11 filesMDMachine Learning8 filesMDNeural Networks12 filesMDNLP7 filesMDTokenization12 filesMDEmbeddings7 filesMDPositional Information6 filesMDAttention12 filesMDMulti-Head Attention6 filesMDTransformer10 filesMDLanguage Modeling6 filesMDText Generation7 filesMDStopping Generation6 filesMDLLM Architecture9 filesMDContext Window6 filesMDLLM Training15 filesMDBuild a Small LLM10 filesMDModel Precision5 filesMDQuantization14 filesMDGPU & Memory7 filesMDKV Cache5 filesMDFine-Tuning6 filesMDLoRA7 filesMDAlignment6 filesMDRAG12 filesMDVector Search10 filesMDFunction Calling7 filesMDAgents8 filesMDInference9 filesMDInference Optimization4 filesMDDistributed Training7 filesMDMixture of Experts6 filesMDMultimodal LLMs7 filesMDReasoning Models6 filesMDEvaluation10 filesMDLLM Systems9 filesMDLLM Research10 files

Course progress

0%

0% of 38 sections

LLM Learning Hub

workspace/llm-course/home

← LLM Systems

Model Serving

Model serving is the deployment of a trained LLM behind an API that handles inference requests, batching, caching, and scaling to serve users in production.

What is Model Serving?

Model serving is the deployment of a trained LLM behind an API that handles inference requests, batching, caching, and scaling to serve users in production.

Model serving is the deployment of a trained LLM behind an API that handles inference requests, batching, caching, and scaling to serve users in production.

Where is it used?

vLLM, TensorRT-LLM, TGI, and Triton serve Llama/Mistral in production; OpenAI and Anthropic run custom serving stacks; `TextGenerationInference` powers HF Inference Endpoints.

How to build it

Serve Llama-3-8B with `python -m vllm.entrypoints.openai.api_server --model meta-llama/Meta-Llama-3-8B-Instruct --port 8000`, then call `POST /v1/chat/completions` with a curl request.

Back to LLM Systems