Backpropagation

Advanced

Introduction

Gradient descent needs the derivative of the loss with respect to every weight. A modern network has millions of weights, and nudging each one separately to estimate its derivative would be hopelessly slow. Backpropagation computes all of them at once, by running the chain rule backwards through the calculation. It is the algorithm that made deep learning practical, and every framework implements it as automatic differentiation.

How it works

Any calculation can be drawn as a computation graph: inputs and weights at the bottom, operations in the middle, the loss at the top. For example, n = x₁w₁ + x₂w₂ + b and out = tanh(n).

Each operation knows its local gradient, the derivative of its output with respect to each of its inputs:

  • addition z = a + b passes the gradient through unchanged: ∂z/∂a = ∂z/∂b = 1;
  • multiplication z = a × b swaps the other input: ∂z/∂a = b, ∂z/∂b = a;
  • tanh has derivative 1 − tanh²; ReLU has derivative 1 where its input is positive and 0 elsewhere.

The chain rule says the gradient of the output with respect to any value is the gradient arriving from above multiplied by the local gradient. So backpropagation:

  1. runs the calculation forward, keeping every intermediate value;
  2. sets the gradient of the output with respect to itself to 1;
  3. visits the operations in reverse order, and for each one multiplies the incoming gradient by its local gradients and adds the result to its inputs' gradients.

The adding matters: if a value is used twice, as x in y = x × x, its gradient is the sum of the contributions from both uses, 2x.

For a network layer a = f(W x + b) with loss L, the same rules give ∂L/∂W = (∂L/∂z) xᵀ, ∂L/∂b = ∂L/∂z and ∂L/∂x = Wᵀ (∂L/∂z), where ∂L/∂z = ∂L/∂a × f′(z). Repeating this layer by layer, from the output back to the input, gives every weight's gradient.

Syntax

def __mul__(self, other):
    out = Value(self.data * other.data, (self, other))
    def backward():
        self.grad += other.data * out.grad    # local gradient × incoming
        other.grad += self.data * out.grad
    out._backward = backward
    return out

Example

import math


class Value:
    def __init__(self, data, parents=(), backward=lambda: None, label=""):
        self.data, self.grad = data, 0.0
        self.parents, self._backward, self.label = parents, backward, label

    def __add__(self, other):
        out = Value(self.data + other.data, (self, other))

        def backward():
            self.grad += out.grad
            other.grad += out.grad

        out._backward = backward
        return out

    def __mul__(self, other):
        out = Value(self.data * other.data, (self, other))

        def backward():
            self.grad += other.data * out.grad
            other.grad += self.data * out.grad

        out._backward = backward
        return out

    def tanh(self):
        t = round(math.tanh(self.data), 12)  # rounded, so every computer gives the same digits
        out = Value(t, (self,))

        def backward():
            self.grad += (1 - t * t) * out.grad

        out._backward = backward
        return out

    def backpropagate(self):
        order = []

        def visit(v):
            # A value is added only after all its parents, so one already in order is done.
            if all(v is not done for done in order):
                for p in v.parents:
                    visit(p)
                order.append(v)

        visit(self)
        self.grad = 1.0
        for v in reversed(order):
            v._backward()


x1, x2 = Value(1.5, label="x1"), Value(-0.5, label="x2")
w1, w2, b = Value(0.8, label="w1"), Value(-1.2, label="w2"), Value(0.1, label="b")
n = x1 * w1 + x2 * w2 + b
out = n.tanh()
out.backpropagate()
print(f"n = {n.data:.4f}, out = tanh(n) = {out.data:.4f}")
for v in [w1, w2, b, x1, x2]:
    print(f"d out / d {v.label} = {v.grad:.4f}")

h = 1e-6
bumped = round(math.tanh(1.5 * (0.8 + h) + (-0.5) * (-1.2) + 0.1), 12)
print(f"numerical check for w1: {(bumped - out.data) / h:.4f}")
Open in compiler

Line by line

  1. Value stores a number, its gradient and the values it was computed from.
  2. Each operation creates a new Value and a backward function that applies its local gradients.
  3. backpropagate orders the graph so that each value comes after everything it depends on, then runs the backward functions from the output down.
  4. The output tanh(1.9) = 0.9562 changes slowly here, because tanh is nearly flat, so every gradient is small.
  5. The numerical check for w1 agrees with the gradient from backpropagation.

Expected result

n = 1.9000, out = tanh(n) = 0.9562
d out / d w1 = 0.1284
d out / d w2 = -0.0428
d out / d b = 0.0856
d out / d x1 = 0.0685
d out / d x2 = -0.1027
numerical check for w1: 0.1284

In the real world

PyTorch's autograd and JAX record operations exactly like this Value class, but on whole arrays, and run the backward pass on graphics processors. Training a large model spends roughly twice as much time on the backward pass as on the forward pass. Understanding backpropagation explains problems such as vanishing gradients, which the following lessons address.

Common mistakes

  • Setting gradients instead of adding to them, which loses contributions when a value is used twice.
  • Forgetting to reset gradients to zero before the next backward pass, so they keep growing.
  • Visiting operations in the wrong order, before the gradient arriving from above is complete.
  • Using the activation's output where its input is needed in the derivative, or the other way round.

Best practices

  • Check every new operation's backward function with a numerical gradient.
  • Work through a two-operation example by hand before writing code.
  • Keep the forward values: the backward pass needs them.

Summary

Backpropagation applies the chain rule from the output back to the inputs. Each operation multiplies the incoming gradient by its local gradient and adds the result to its inputs' gradients, so values used several times collect all their contributions. It computes every gradient of a network in one backward pass.

Quick check

1. What is the local gradient of z = a × b with respect to a?
2. Why are gradients added rather than overwritten?
3. In which order does backpropagation visit operations?
4. What is the derivative of ReLU at a negative input?

Key takeaways

  • Each operation only needs its local gradient; the chain rule combines them.
  • Gradients flow backwards and add up where a value feeds several operations.
  • Backpropagation computes all gradients in about the same time as one forward pass.

Practice problems for this lesson

My notes

Write down what you want to remember from this lesson. Notes are saved on this device as you type, and appear on your dashboard.