Vision-Language Models
Vision-Language Models (VLMs) are multimodal models specialised in understanding images and text together, answering questions about images, captioning, and reading documents.
What is Vision-Language Models?
Vision-Language Models (VLMs) are multimodal models specialised in understanding images and text together, answering questions about images, captioning, and reading documents.
Vision-Language Models (VLMs) are multimodal models specialised in understanding images and text together, answering questions about images, captioning, and reading documents.
Where is it used?
LLaVA, Qwen2-VL, InternVL, and GPT-4o are VLMs; they power visual chat, OCR-free document understanding, and robotics perception; HF `VLChat` pipelines serve them.
How to build it
Build LLaVA-style: project CLIP image features with an MLP into the LLM embedding space, prepend them as pseudo-tokens to the text prompt, and fine-tune with `Trainer` on instruction data.