LibraryProgress

Generated Files

llm-course/home

Topics

MDComputer & Python Foundations7 filesMDMathematics11 filesMDMachine Learning8 filesMDNeural Networks12 filesMDNLP7 filesMDTokenization12 filesMDEmbeddings7 filesMDPositional Information6 filesMDAttention12 filesMDMulti-Head Attention6 filesMDTransformer10 filesMDLanguage Modeling6 filesMDText Generation7 filesMDStopping Generation6 filesMDLLM Architecture9 filesMDContext Window6 filesMDLLM Training15 filesMDBuild a Small LLM10 filesMDModel Precision5 filesMDQuantization14 filesMDGPU & Memory7 filesMDKV Cache5 filesMDFine-Tuning6 filesMDLoRA7 filesMDAlignment6 filesMDRAG12 filesMDVector Search10 filesMDFunction Calling7 filesMDAgents8 filesMDInference9 filesMDInference Optimization4 filesMDDistributed Training7 filesMDMixture of Experts6 filesMDMultimodal LLMs7 filesMDReasoning Models6 filesMDEvaluation10 filesMDLLM Systems9 filesMDLLM Research10 files

Course progress

0%

0% of 38 sections

LLM Learning Hub

workspace/llm-course/home

← Multimodal LLMs

Vision Encoder

The vision encoder is the image backbone (usually a ViT) that extracts patch-level features from an image, which are then projected and fed into the language model as visual tokens.

What is Vision Encoder?

The vision encoder is the image backbone (usually a ViT) that extracts patch-level features from an image, which are then projected and fed into the language model as visual tokens.

The vision encoder is the image backbone (usually a ViT) that extracts patch-level features from an image, which are then projected and fed into the language model as visual tokens.

Where is it used?

LLaVA uses CLIP-ViT-L/14, Qwen2-VL uses a custom ViT with dynamic resolution, and InternVL uses InternViT; SigLIP is the encoder for Pixtral and Idefics3.

How to build it

Load `ViTModel.from_pretrained("google/vit-base-patch16-224")`, pass `pixel_values`, and take `outputs.last_hidden_state` (shape `(1, n_patches, hidden)`) as visual token features.

Back to Multimodal LLMs