fa25-cs540 cs540
Lecture 13: Neural Networks III Lecture 13: Neural Networks III
Content locked
The catalog is public. Reading this content requires approved access.
Sign in to readLecture 13: Neural Networks III
Introduction
Neural Networks as a Computational Graph
Deep Neural Networks (DNNs)
Neural Networks as Variables + Operations
Forward and Backward Propagation
Modern Treatment of Backpropagation
Backpropagation in PyTorch
Numerical Stability
Gradients for Neural Networks
Two Major Issues
Issues with Gradient Exploding
Gradient Vanishing with Sigmoid
Issues with Gradient Vanishing
Solutions for Numerical Stability
Practice Problems: Gradient Descent and Stability
Generalization and Regularization
Training Error vs Generalization Error
Underfitting and Overfitting
Model Capacity
Classical View of Model Complexity
A Modern View: Double Descent
At the Frontier: Bigger is Better
Estimating Neural Network Capacity
Data Complexity
Practice Problems: Overfitting
Regularization Techniques
Weight Decay (L2 Regularization)
Illustration of Weight Decay Effect
Dropout
Applying Dropout
Dropout Effectiveness
Practice Problem: Dropout
Key Takeaways
Suggested Reading