Making training work
Initialization and vanishing gradients, batch and layer normalization, and regularization against overfitting.
About 5 hours of study
Learning objectives
- Choose initialization scales
- Apply batch and layer normalization
- Detect and reduce overfitting
Lessons
- 1
Initialization and vanishing gradients Advanced
Why the starting values of the weights matter: signals and gradients shrink or explode as they pass through many layers. Use Xavier and He initialization, and see how activation functions affect gradient flow.
- 2
Batch normalization and layer normalization Advanced
Keep the values inside a network on a steady scale by normalizing them: subtract a mean, divide by a standard deviation, then rescale with learned parameters. Compare batch norm and layer norm, and handle training and inference.
- 3
Overfitting and regularization Advanced
A model that memorizes its training data fails on new data. Detect overfitting with a validation set, and prevent it with L2 weight decay, dropout and early stopping.
Module practice
- Deep learning: Initialization scalesEasy
- Deep learning: Watch a gradient vanish or explodeMedium
- Deep learning: Activation statistics per layerHard
- Deep learning: Standardize a listEasy
- Deep learning: A batch normalization layerMedium
- Deep learning: Batch norm in training and at inferenceHard
- Deep learning: L2 penalty and weight decayEasy
- Deep learning: Inverted dropoutMedium
- Deep learning: Early stopping with patienceHard