Training networks

Multi-layer networks and the training loop, softmax classifiers with cross-entropy, and optimizers from momentum to Adam.

About 5 hours of study

Learning objectives

Lessons

  1. 1

    Training a multi-layer network Intermediate

    Put neurons, losses, gradient descent and backpropagation together to train a network with a hidden layer. Solve XOR, which no single neuron can, and understand epochs, batches and why hidden layers matter.

  2. 2

    Softmax and multi-class classification Intermediate

    Turn a network's raw scores into probabilities over many classes with softmax, train with cross-entropy, and use the simple gradient that results. Compute softmax safely for large numbers.

  3. 3

    Optimizers: momentum, RMSProp and Adam Advanced

    Plain gradient descent zigzags in narrow valleys and crawls on flat ground. Learn how momentum, adaptive learning rates and Adam fix this, use mini-batches, and schedule the learning rate during training.

Module practice

Module quiz

1. How many parameters does a layer with 3 inputs and 4 neurons have?
2. What do softmax outputs add up to?
3. Why can't a single neuron learn XOR?
4. What is the gradient of softmax cross-entropy with respect to the logits?
5. What does momentum keep between steps?
6. Why does Adam use bias correction?

‹ Neural network basicsMaking training work ›