Streaming
Streaming inference returns tokens to the client incrementally as they are generated (via SSE or WebSocket), improving perceived latency by showing text progressively rather than waiting for full completion.
What is Streaming?
Streaming inference returns tokens to the client incrementally as they are generated (via SSE or WebSocket), improving perceived latency by showing text progressively rather than waiting for full completion.
Streaming inference returns tokens to the client incrementally as they are generated (via SSE or WebSocket), improving perceived latency by showing text progressively rather than waiting for full completion.
Where is it used?
ChatGPT, Claude, and vLLM's OpenAI-compatible server support `stream=True`; HF `TextIteratorStreamer` enables streaming in custom serving scripts.
How to build it
Use `transformers.TextIteratorStreamer(tokenizer)` with `Thread` and `model.generate(streamer=streamer, ...)`, then `for text in streamer: print(text, end="")` to stream tokens.