Cross-Modal Representation
Cross-modal representation is the shared embedding space where vectors from different modalities (text, image, audio) are aligned so that semantically related content is close regardless of modality.
What is Cross-Modal Representation?
Cross-modal representation is the shared embedding space where vectors from different modalities (text, image, audio) are aligned so that semantically related content is close regardless of modality.
Cross-modal representation is the shared embedding space where vectors from different modalities (text, image, audio) are aligned so that semantically related content is close regardless of modality.
Where is it used?
CLIP's contrastive image-text alignment and ImageBind's six-modality shared space are foundational; LLaVA's projection layer aligns image features into the text LLM's space.
How to build it
Train a CLIP-like model with `InfoNCE` contrastive loss between image and text embeddings using `transformers.CLIPModel`, or fine-tune a projection layer with `nn.CosineEmbeddingLoss`.