Video Understanding
Video understanding models process sequences of frames plus audio/text to answer questions about video content, recognise actions, and summarise events over time.
What is Video Understanding?
Video understanding models process sequences of frames plus audio/text to answer questions about video content, recognise actions, and summarise events over time.
Video understanding models process sequences of frames plus audio/text to answer questions about video content, recognise actions, and summarise events over time.
Where is it used?
Gemini 1.5 Pro, GPT-4o, and Qwen2-VL (video) handle video QA; Video-LLaVA and LLaVA-NeXT-Video are open models; they sample frames, embed each, and feed them as a temporal sequence.
How to build it
Sample N frames from a video with `decord`, embed each frame with CLIP, project into LLM space, prepend as N visual tokens with positional encodings, and generate a caption.