Training networks
Multi-layer networks and the training loop, softmax classifiers with cross-entropy, and optimizers from momentum to Adam.
About 5 hours of study
Learning objectives
- Train a network with a hidden layer
- Train a multi-class classifier
- Use momentum, Adam and learning-rate schedules
Lessons
- 1
Training a multi-layer network Intermediate
Put neurons, losses, gradient descent and backpropagation together to train a network with a hidden layer. Solve XOR, which no single neuron can, and understand epochs, batches and why hidden layers matter.
- 2
Softmax and multi-class classification Intermediate
Turn a network's raw scores into probabilities over many classes with softmax, train with cross-entropy, and use the simple gradient that results. Compute softmax safely for large numbers.
- 3
Optimizers: momentum, RMSProp and Adam Advanced
Plain gradient descent zigzags in narrow valleys and crawls on flat ground. Learn how momentum, adaptive learning rates and Adam fix this, use mini-batches, and schedule the learning rate during training.
Module practice
- Deep learning: Count the parametersEasy
- Deep learning: Full-batch trainingMedium
- Deep learning: Gradients averaged over a batchHard
- Deep learning: A stable softmaxEasy
- Deep learning: Cross-entropy and its gradientMedium
- Deep learning: One step of softmax regressionHard
- Deep learning: Gradient descent with momentumEasy
- Deep learning: Learning-rate schedulesMedium
- Deep learning: The Adam optimizerHard