Neural network basics

Neurons and activation functions, losses and gradient descent, and backpropagation through computation graphs.

About 5 hours of study

Learning objectives

Lessons

  1. 1

    Neurons, layers and activation functions Intermediate

    The building block of every neural network: a weighted sum of inputs plus a bias, passed through an activation function. Compute neurons and whole layers by hand and in Python, and compare sigmoid, tanh and ReLU.

  2. 2

    Loss functions and gradient descent Intermediate

    Measure how wrong a model is with a loss function, and reduce the loss by repeatedly stepping the parameters downhill along the gradient. Use mean squared error and cross-entropy, choose a learning rate, and check gradients numerically.

  3. 3

    Backpropagation Advanced

    Compute every gradient in a network efficiently by applying the chain rule backwards through the computation. Draw computation graphs, propagate gradients by hand, and build a tiny automatic-differentiation engine.

Module practice

Module quiz

1. What does a neuron compute before its activation?
2. What is ReLU(−3)?
3. Which way does gradient descent move a parameter?
4. What happens when the learning rate is far too large?
5. What is the local gradient of z = a × b with respect to a?
6. Why does backpropagation add gradient contributions?

Training networks ›