Neural network basics
Neurons and activation functions, losses and gradient descent, and backpropagation through computation graphs.
About 5 hours of study
Learning objectives
- Compute neurons and layers
- Minimize a loss with gradient descent
- Compute gradients with backpropagation
Lessons
- 1
Neurons, layers and activation functions Intermediate
The building block of every neural network: a weighted sum of inputs plus a bias, passed through an activation function. Compute neurons and whole layers by hand and in Python, and compare sigmoid, tanh and ReLU.
- 2
Loss functions and gradient descent Intermediate
Measure how wrong a model is with a loss function, and reduce the loss by repeatedly stepping the parameters downhill along the gradient. Use mean squared error and cross-entropy, choose a learning rate, and check gradients numerically.
- 3
Backpropagation Advanced
Compute every gradient in a network efficiently by applying the chain rule backwards through the computation. Draw computation graphs, propagate gradients by hand, and build a tiny automatic-differentiation engine.
Module practice
- Deep learning: Compute one neuronEasy
- Deep learning: A dense layerMedium
- Deep learning: Forward pass through a networkHard
- Deep learning: Two loss functionsEasy
- Deep learning: Fit a line by gradient descentMedium
- Deep learning: Check a gradient numericallyHard
- Deep learning: Backpropagate by handEasy
- Deep learning: Gradients of one neuronMedium
- Deep learning: One training step of a networkHard