Neural Networks Taxonomy - 15. Multimodal and Vision-Language

15. Multimodal & Vision-Language

A multimodal network handles more than one kind of input at once — most often images together with text. The standard recipe, in Fig 17, encodes each modality with its own network — a convolutional or vision Transformer for images, a Transformer for text — and then fuses or aligns the two representations into a shared space.

Origins and rise

CLIP aligned images and captions with a contrastive objective — the same metric-learning idea applied across modalities — and unlocked zero-shot classification from natural-language labels.^[208] Flamingo then showed few-shot visual question answering by feeding image features into a frozen language model.^[209] The same encoders feed the generative text-to-image systems that turn a prompt into a picture.

Image-text alignment

CLIP's contrastive recipe^[208] was quickly scaled and refined. ALIGN showed that a noisy billion-scale web dataset could stand in for careful curation,^[210] LiT found that locking a pre-trained image encoder while tuning only the text tower gives stronger zero-shot transfer,^[211] and SigLIP swapped the softmax contrastive loss for a simple pairwise sigmoid that trains well even at small batch sizes.^[212]

Fusion encoders for question answering

Before contrastive pre-training, visual question answering was driven by attention over detected objects: the Bottom-Up and Top-Down model attended to region features from an object detector.^[213] The BERT era then produced Transformer encoders that fuse the two modalities directly: ViLBERT used two streams joined by cross-attention,^[214] LXMERT a comparable cross-modality encoder,^[215] and UNITER a single stream that concatenates image regions with word tokens.^[216]

Generative vision-language models

The next wave connected vision encoders to generative language models so the output is free-form text. BLIP bootstrapped noisy web captions to pre-train for both understanding and generation,^[217] and BLIP-2 bridged a frozen image encoder to a frozen large language model through a lightweight querying transformer.^[218] Flamingo interleaved image and text tokens for few-shot prompting,^[209] while PaLI jointly scaled the vision and language towers across many languages.^[219]

Visual grounding

Grounding models tie phrases to specific image regions. MDETR extended the DETR detector^[23] to condition detection on a free-text query,^[220] and Grounding DINO married a strong detector with grounded pre-training for open-set detection from arbitrary prompts.^[221] Kosmos-2 folded grounding into a multimodal language model, letting it point to regions as it generates text.^[222]

Document understanding

A specialised branch reads text-rich images such as forms, receipts, and screenshots. LayoutLMv3 jointly masks text and image patches to learn layout-aware representations,^[223] while Donut reads documents OCR-free, decoding structured output straight from pixels.^[224] Pix2Struct pre-trained by parsing masked webpage screenshots, transferring to charts, user interfaces, and diagrams.^[225]

Unified any-task models

The frontier folds every task into one sequence-to-sequence model. CoCa trained a single network under both contrastive and captioning objectives,^[226] and Unified-IO cast vision, language, and dense prediction tasks into one shared token vocabulary.^[227] BEiT-3 treated images as another language to pre-train a shared backbone across vision and vision-language tasks,^[228] and ImageBind learned a single embedding space binding six modalities using only image-paired data.^[229]

Applications

• Text-to-image retrieval and zero-shot classification.
• Visual question answering, captioning, and document understanding.
• Open-vocabulary detection and multimodal assistants.

Strengths and limitations