sp26-cs639 lectureattentionflash-attentionstate-space-modelsmambacs639
Lecture 11: Attention Variations Lecture 11: Attention Variations
Content locked
The catalog is public. Reading this content requires approved access.
Sign in to readLecture 11: Attention Variations
Review: Decoder-Only Models: GPT
Autoregressive Next Token Prediction
From GPT-2 to GPT-3
Open Source: Llama 3.1
Attention: Variations
Motivation: Complexity
Motivation: Biases
Sparse Attention Mechanisms
Variation 1: Sliding Window
Variation 2: Local + Global
Variation 3: Block Attention
Variation 4: Learned Attention
Beyond Sparsity
Hardware Considerations
Flash Attention: Basic Idea
Flash Attention: Tiling
Flash Attention: Recomputing
Flash Attention: Tradeoffs
Beyond Attention
Attention Alternatives
State-Space Model
State-Space Model: Discrete Form
State-Space Model: Convolutional Form
S4: Structured State Space Models
Using SSMs as Layers
S4 Results: The Good and the Bad
Mamba Model