Introduction
Gradient descent needs the derivative of the loss with respect to every weight. A modern network has millions of weights, and nudging each one separately to estimate its derivative would be hopelessly slow. Backpropagation computes all of them at once, by running the chain rule backwards through the calculation. It is the algorithm that made deep learning practical, and every framework implements it as automatic differentiation.
How it works
Any calculation can be drawn as a computation graph: inputs and weights at the bottom, operations in the middle, the loss at the top. For example, n = x₁w₁ + x₂w₂ + b and out = tanh(n).
Each operation knows its local gradient, the derivative of its output with respect to each of its inputs:
- addition z = a + b passes the gradient through unchanged: ∂z/∂a = ∂z/∂b = 1;
- multiplication z = a × b swaps the other input: ∂z/∂a = b, ∂z/∂b = a;
- tanh has derivative 1 − tanh²; ReLU has derivative 1 where its input is positive and 0 elsewhere.
The chain rule says the gradient of the output with respect to any value is the gradient arriving from above multiplied by the local gradient. So backpropagation:
- runs the calculation forward, keeping every intermediate value;
- sets the gradient of the output with respect to itself to 1;
- visits the operations in reverse order, and for each one multiplies the incoming gradient by its local gradients and adds the result to its inputs' gradients.
The adding matters: if a value is used twice, as x in y = x × x, its gradient is the sum of the contributions from both uses, 2x.
For a network layer a = f(W x + b) with loss L, the same rules give ∂L/∂W = (∂L/∂z) xᵀ, ∂L/∂b = ∂L/∂z and ∂L/∂x = Wᵀ (∂L/∂z), where ∂L/∂z = ∂L/∂a × f′(z). Repeating this layer by layer, from the output back to the input, gives every weight's gradient.
Syntax
def __mul__(self, other):
out = Value(self.data * other.data, (self, other))
def backward():
self.grad += other.data * out.grad # local gradient × incoming
other.grad += self.data * out.grad
out._backward = backward
return out
Example
import math
class Value:
def __init__(self, data, parents=(), backward=lambda: None, label=""):
self.data, self.grad = data, 0.0
self.parents, self._backward, self.label = parents, backward, label
def __add__(self, other):
out = Value(self.data + other.data, (self, other))
def backward():
self.grad += out.grad
other.grad += out.grad
out._backward = backward
return out
def __mul__(self, other):
out = Value(self.data * other.data, (self, other))
def backward():
self.grad += other.data * out.grad
other.grad += self.data * out.grad
out._backward = backward
return out
def tanh(self):
t = round(math.tanh(self.data), 12) # rounded, so every computer gives the same digits
out = Value(t, (self,))
def backward():
self.grad += (1 - t * t) * out.grad
out._backward = backward
return out
def backpropagate(self):
order = []
def visit(v):
# A value is added only after all its parents, so one already in order is done.
if all(v is not done for done in order):
for p in v.parents:
visit(p)
order.append(v)
visit(self)
self.grad = 1.0
for v in reversed(order):
v._backward()
x1, x2 = Value(1.5, label="x1"), Value(-0.5, label="x2")
w1, w2, b = Value(0.8, label="w1"), Value(-1.2, label="w2"), Value(0.1, label="b")
n = x1 * w1 + x2 * w2 + b
out = n.tanh()
out.backpropagate()
print(f"n = {n.data:.4f}, out = tanh(n) = {out.data:.4f}")
for v in [w1, w2, b, x1, x2]:
print(f"d out / d {v.label} = {v.grad:.4f}")
h = 1e-6
bumped = round(math.tanh(1.5 * (0.8 + h) + (-0.5) * (-1.2) + 0.1), 12)
print(f"numerical check for w1: {(bumped - out.data) / h:.4f}")Line by line
Valuestores a number, its gradient and the values it was computed from.- Each operation creates a new
Valueand abackwardfunction that applies its local gradients. backpropagateorders the graph so that each value comes after everything it depends on, then runs the backward functions from the output down.- The output tanh(1.9) = 0.9562 changes slowly here, because tanh is nearly flat, so every gradient is small.
- The numerical check for w1 agrees with the gradient from backpropagation.
Expected result
n = 1.9000, out = tanh(n) = 0.9562
d out / d w1 = 0.1284
d out / d w2 = -0.0428
d out / d b = 0.0856
d out / d x1 = 0.0685
d out / d x2 = -0.1027
numerical check for w1: 0.1284
In the real world
PyTorch's autograd and JAX record operations exactly like this Value class, but on whole arrays, and run the backward pass on graphics processors. Training a large model spends roughly twice as much time on the backward pass as on the forward pass. Understanding backpropagation explains problems such as vanishing gradients, which the following lessons address.
Common mistakes
- Setting gradients instead of adding to them, which loses contributions when a value is used twice.
- Forgetting to reset gradients to zero before the next backward pass, so they keep growing.
- Visiting operations in the wrong order, before the gradient arriving from above is complete.
- Using the activation's output where its input is needed in the derivative, or the other way round.
Best practices
- Check every new operation's backward function with a numerical gradient.
- Work through a two-operation example by hand before writing code.
- Keep the forward values: the backward pass needs them.
Summary
Backpropagation applies the chain rule from the output back to the inputs. Each operation multiplies the incoming gradient by its local gradient and adds the result to its inputs' gradients, so values used several times collect all their contributions. It computes every gradient of a network in one backward pass.
Quick check
Key takeaways
- Each operation only needs its local gradient; the chain rule combines them.
- Gradients flow backwards and add up where a value feeds several operations.
- Backpropagation computes all gradients in about the same time as one forward pass.
Practice problems for this lesson
My notes
Write down what you want to remember from this lesson. Notes are saved on this device as you type, and appear on your dashboard.