Deep Learning
Neurons, backpropagation, optimizers, normalization, CNNs, RNNs, embeddings, attention and transformers, every network built from scratch in plain Python.
- Level
- Advanced
- Time
- About 25 hours
- Modules
- 5
- Lessons
- 15
- Exercises
- 45
Deep learning trains networks of simple units to recognize images, understand speech and generate text. This course builds it from the ground up: a single neuron, losses and gradient descent, backpropagation, multi-layer networks and softmax classifiers, the optimizers and normalization layers that make training work, convolutional and recurrent networks, and finally embeddings, attention and the transformer.
There are no libraries and nothing is hidden. Every forward pass, gradient and update is plain Python that runs in your browser, on networks small enough to follow by hand. When you later use PyTorch or TensorFlow, you will know what each layer and each line of the training loop is doing.
What you will learn
- Compute neurons, layers and whole networks by hand and in code
- Derive and implement backpropagation and gradient descent
- Train classifiers with softmax, cross-entropy and modern optimizers
- Keep training stable with initialization, normalization and regularization
- Implement convolutions, pooling and recurrent cells, with their gradients
- Implement embeddings, attention and the parts of a transformer
Skills you will gain
- Forward and backward passes
- Optimization and training loops
- Initialization, normalization and regularization
- Convolutional and recurrent networks
- Embeddings, attention and transformers
Before you start
- Python: functions, lists, loops and classes (the Python course)
- Comfort with algebra; derivatives are explained as they are needed
- The Machine Learning course helps but is not required
Where it is used
- Image classification and detection
- Speech and time-series models
- Language models and text generation
- Recommendation and search with embeddings
Modules
Module 1: Neural network basics
Neurons and activation functions, losses and gradient descent, and backpropagation through computation graphs.
- 1
Neurons, layers and activation functions Intermediate
The building block of every neural network: a weighted sum of inputs plus a bias, passed through an activation function. Compute neurons and whole layers by hand and in Python, and compare sigmoid, tanh and ReLU.
- 2
Loss functions and gradient descent Intermediate
Measure how wrong a model is with a loss function, and reduce the loss by repeatedly stepping the parameters downhill along the gradient. Use mean squared error and cross-entropy, choose a learning rate, and check gradients numerically.
- 3
Backpropagation Advanced
Compute every gradient in a network efficiently by applying the chain rule backwards through the computation. Draw computation graphs, propagate gradients by hand, and build a tiny automatic-differentiation engine.
Module 2: Training networks
Multi-layer networks and the training loop, softmax classifiers with cross-entropy, and optimizers from momentum to Adam.
- 1
Training a multi-layer network Intermediate
Put neurons, losses, gradient descent and backpropagation together to train a network with a hidden layer. Solve XOR, which no single neuron can, and understand epochs, batches and why hidden layers matter.
- 2
Softmax and multi-class classification Intermediate
Turn a network's raw scores into probabilities over many classes with softmax, train with cross-entropy, and use the simple gradient that results. Compute softmax safely for large numbers.
- 3
Optimizers: momentum, RMSProp and Adam Advanced
Plain gradient descent zigzags in narrow valleys and crawls on flat ground. Learn how momentum, adaptive learning rates and Adam fix this, use mini-batches, and schedule the learning rate during training.
Module 3: Making training work
Initialization and vanishing gradients, batch and layer normalization, and regularization against overfitting.
- 1
Initialization and vanishing gradients Advanced
Why the starting values of the weights matter: signals and gradients shrink or explode as they pass through many layers. Use Xavier and He initialization, and see how activation functions affect gradient flow.
- 2
Batch normalization and layer normalization Advanced
Keep the values inside a network on a steady scale by normalizing them: subtract a mean, divide by a standard deviation, then rescale with learned parameters. Compare batch norm and layer norm, and handle training and inference.
- 3
Overfitting and regularization Advanced
A model that memorizes its training data fails on new data. Detect overfitting with a validation set, and prevent it with L2 weight decay, dropout and early stopping.
Module 4: Convolutional and recurrent networks
Convolutions with padding, stride and gradients, pooling and complete CNNs, and recurrent networks, backpropagation through time and LSTMs.
- 1
Convolutions Advanced
Slide a small kernel of weights across an image to detect local patterns such as edges. Compute convolutions by hand and in code, control the output size with padding and stride, and find the gradients a convolution layer needs for training.
- 2
Pooling and convolutional networks Advanced
Build a whole convolutional network: convolution layers with ReLU, pooling layers that shrink feature maps, then flattening and dense layers for the final decision. Track shapes and parameter counts through the network.
- 3
Recurrent networks and LSTMs Advanced
Process sequences such as text, sound and time series with a hidden state carried from step to step. Run a recurrent network, backpropagate through time, see why gradients vanish over long sequences, and how LSTM gates fix it.
Module 5: Attention and transformers
Embeddings and similarity, scaled dot-product and causal self-attention, and the transformer with positional encodings, parameter counts and decoding.
- 1
Embeddings Advanced
Represent words and other symbols as dense vectors whose directions carry meaning. Compare embeddings with cosine similarity, find nearest neighbours, solve analogies with vector arithmetic, and see how embedding tables are learned.
- 2
Attention Advanced
Let every position in a sequence look at every other position and take a weighted average of what it finds. Compute scaled dot-product attention, use self-attention and causal masks, and see why attention replaced recurrence.
- 3
Transformers Advanced
Assemble the architecture behind modern language models: token embeddings plus positional encodings, then blocks of multi-head self-attention and feed-forward layers with residual connections and layer normalization, and finally next-token prediction and decoding.
Projects
- A digit classifier from scratch Build and train a neural network in plain Python that recognizes small pixel images of the digits 0 to 9, and measure how well it works.
- A tiny language model Train a character-level model on a short text and use it to generate new text, first with a simple next-character model and then with a small attention layer.
What to learn next
- Machine Learning Machine learning from scratch in plain Python: regression, gradient descent, regularization, k-NN, logistic regression, metrics, trees, ensembles and k-means.
- Artificial Intelligence Search and A*, game playing, constraint solving, logic, Bayesian networks, hidden Markov models, MDPs and Q-learning, each built in Python.