Lecture 12: Multimodal Models I

Content locked

The catalog is public. Reading this content requires approved access.

Review: Attention Variations

Review: Hardware Considerations

Flash Attention: Hardware-Aware Attention

Flash Attention: Basic Idea

Flash Attention: Tiling

Flash Attention: Recomputing

Flash Attention: Tradeoffs

Attention Alternatives

State-Space Model

Discrete Form

Convolutional Form

Back to SSM Picture

Using SSMs as Layers

S4 Results: The Good and the Bad

Multimodal Models Intro

Short History of Multimodal Models

Making LLMs Multimodal

VisualBERT Training

How Do We Get Image Embeddings?

Put It Together: ViLT

Variations

Datasets

VLM Variations and Types

Contrastive Vision-Language Models

Contrastive Training

CLIP

How to Use CLIP?

FLAVA

Few-Shot VLMs

Few-Shot VLMs: Flamingo

Visual Instructions: LLaVA

Closed-Source Multimodal Models