sp26-cs639 lecturescaling-lawspre-trainingdata-efficiencycs639
Lecture 6: Guest Lecture — The Science of Foundation Models for Science Lecture 6: Guest Lecture — The Science of Foundation Models for Science
Content locked
The catalog is public. Reading this content requires approved access.
Sign in to readLecture 6: Guest Lecture — The Science of Foundation Models for Science
Overview
Principled Foundation Model Design
Pre-Training
Pre-Training Scaling Laws
Compute Optimal Scaling (Chinchilla)
Skill-Dependent Scaling
Effect of Data Distribution
Practical Implications
Other Work in Principled FM Design
Data Efficiency
Motivation: Humans Learn in Order
Skilllet Algorithm
Assumptions
How Skilllet Works
Results
Other Work on Data Efficiency
Adaptation to Science
MELGAN: Evaluating Agents on ML Research Tasks
Other Work
Future Directions