Multimodal Models
Multimodal models process and generate content across multiple modalities — text, images, audio, video — within a single model, enabling cross-modal understanding and generation.
What is Multimodal Models?
Multimodal models process and generate content across multiple modalities — text, images, audio, video — within a single model, enabling cross-modal understanding and generation.
Multimodal models process and generate content across multiple modalities — text, images, audio, video — within a single model, enabling cross-modal understanding and generation.
Where is it used?
GPT-4o, Gemini, and Claude 3.5 Sonnet are multimodal; LLaVA, Qwen-VL, and Pixtral are open-weight multimodal LLMs built by attaching a vision encoder to a text LLM.
How to build it
Use `transformers.LlavaForConditionalGeneration.from_pretrained("llava-hf/llava-1.5-7b-hf")`, pass `{"image": image, "text": prompt}` via the processor, and call `.generate()`.