Making training work

Initialization and vanishing gradients, batch and layer normalization, and regularization against overfitting.

About 5 hours of study

Learning objectives

Lessons

  1. 1

    Initialization and vanishing gradients Advanced

    Why the starting values of the weights matter: signals and gradients shrink or explode as they pass through many layers. Use Xavier and He initialization, and see how activation functions affect gradient flow.

  2. 2

    Batch normalization and layer normalization Advanced

    Keep the values inside a network on a steady scale by normalizing them: subtract a mean, divide by a standard deviation, then rescale with learned parameters. Compare batch norm and layer norm, and handle training and inference.

  3. 3

    Overfitting and regularization Advanced

    A model that memorizes its training data fails on new data. Detect overfitting with a validation set, and prevent it with L2 weight decay, dropout and early stopping.

Module practice

Module quiz

1. What is He initialization's standard deviation for 8 inputs?
2. What does batch norm average over?
3. Why do many stacked sigmoid layers make gradients vanish?
4. What is the sign of overfitting?
5. Why do kept units get divided by p in inverted dropout?
6. What does batch norm use at inference time?

‹ Training networksConvolutional and recurrent networks ›