Week 10: Large Multimodal Models (LMMs)

Introduction

While traditional LLMs operate purely on text, the next frontier in AI is multimodal modeling—systems that understand and generate across multiple data types such as text, images, video, and audio. Models like GPT-4V, Gemini, CLIP, and LLaVA combine visual perception with language understanding, enabling powerful capabilities including image captioning, document understanding, visual question answering (VQA), and even code generation from screenshots.

Goals for the Week

• Understand the architectures behind vision-language and multimodal LLMs
• Learn about cross-modal embedding alignment and fusion techniques (CLIP, ALIGN, contrastive learning)
• Build systems that generate text from images and query visual content using natural language
• Explore real-world applications: VQA, captioning, OCR, visual grounding, and multimodal RAG
• Evaluate multimodal model outputs for relevance, consistency, and hallucination

Learning Guide

Videos

• Building Multimodal Search and RAG — DeepLearning.AI course on multimodal retrieval with Weaviate.

Optional Short Courses:

• Large Multimodal Model Prompting with Gemini — Practical prompting techniques for vision-language models
• Introducing Multimodal Llama 3.2 — Open-source multimodal model deployment

Articles & Reading

Comprehensive Guides:

• Understanding Multimodal LLMs — Sebastian Raschka’s introduction to architectures and techniques
• Multimodality and Large Multimodal Models (LMMs) — Chip Huyen’s overview of the field’s evolution
• A Multimodal World — HuggingFace’s practical guide

Additional Reading

• Multimodal AI: A Guide to open source vision language models
• Exploring Multimodal Large Language Models

Foundational Papers:

• CLIP: Learning Transferable Visual Models¹ — Contrastive language-image pre-training
• Flamingo: Visual Language Model² — Few-shot learning with vision and language
• BLIP-2: Bootstrapping Language-Image Pre-training³ — Efficient vision-language alignment
• LLaVA: Visual Instruction Tuning⁴ — Open-source multimodal instruction following
• GPT-4V System Card — Capabilities and limitations of GPT-4 Vision

Evaluation & Benchmarks:

• VQAv2: Visual Question Answering — Standard VQA benchmark | Paper
• POPE: Polling-based Object Probing⁵ — Hallucination evaluation
• MMBench: Multi-modal Benchmark — Comprehensive evaluation suite
• LLaVA-Bench: Visual Conversation — Instruction-following evaluation

Models & Tools

Commercial APIs:

• OpenAI GPT-4V — Vision capabilities in ChatGPT | Cookbook
• Google Gemini — Multimodal understanding and generation
• Anthropic Claude 3 — Vision analysis and document understanding
• Microsoft Azure Computer Vision — OCR, captioning, object detection

Open-Source Models:

• LLaVA — Visual instruction tuning | GitHub
• BLIP-2 — Efficient vision-language pre-training
• Qwen-VL — Multilingual multimodal understanding
• CogVLM — Open visual expert language model
• Fuyu-8B — Simplified multimodal architecture

Frameworks & Libraries:

• Transformers Vision — HuggingFace multimodal models
• LangChain Multimodal — Chains for vision-language tasks
• LlamaIndex Multimodal — Multimodal RAG and indexing
• Weaviate — Vector database with multimodal support
• OpenCLIP — Open-source CLIP implementations

Evaluation Tools:

• CLIP Score — Image-text alignment metric
• BLEU/METEOR/CIDEr — Captioning metrics
• LMMs-Eval — Unified evaluation framework
• VLMEvalKit — Comprehensive VLM benchmarking

Applications & Use Cases

Document Understanding:

• Donut: Document Understanding Transformer — OCR-free document analysis
• LayoutLMv3 — Multimodal pre-training for document AI
• Nougat — Academic document parsing

Visual Reasoning:

• VisProg: Visual Programming — Compositional visual reasoning
• ViperGPT — Code-based visual query answering
• Visual ChatGPT — Chaining vision models

Video Understanding:

• Video-LLaVA — Video instruction tuning
• VideoChat — Conversational video understanding
• Gemini Video Analysis — Long video comprehension

Programming Practice

Vision-Language Inference:

• Use OpenAI GPT-4V or Gemini to:
  • Generate descriptions of diagrams, charts, and UI mockups
  • Answer questions about images (VQA)
  • Extract structured data from documents (receipts, forms)
  • Solve visual puzzles and reasoning tasks

Open-Source Model Deployment:

• Run inference with LLaVA or BLIP-2:
  • Caption images from diverse domains
  • Rank image-text relevance pairs
  • Build zero-shot image classifier using CLIP
  • Implement visual search over image gallery

Multimodal RAG System:

• Build a document Q&A system:
  • Embed images and text using CLIP or multimodal embeddings
  • Store in vector database (Weaviate, Qdrant, Chroma)
  • Retrieve relevant images based on text queries
  • Generate answers combining visual and textual context

Evaluation & Testing:

• Benchmark on VQAv2 or MMBench datasets
• Measure hallucination using POPE
• Calculate CLIP Score for image-text alignment
• Create custom test sets for domain-specific applications
• Implement safety checks for visual content (NSFW, bias)

---

References