Introduction
A network's weights start out random, so its first predictions are nonsense. Training adjusts the weights so that the predictions improve. To do that automatically, we need two things: a single number that says how wrong the network is, the loss, and a way to change every weight so that the loss goes down, gradient descent. Every deep learning system, from a small classifier to a large language model, is trained this way.
How it works
The loss function compares predictions with targets:
- Mean squared error for numbers: MSE = (1/n) Σ (ŷᵢ − yᵢ)². Big mistakes count much more than small ones.
- Binary cross-entropy for probabilities: −(1/n) Σ [yᵢ log ŷᵢ + (1 − yᵢ) log(1 − ŷᵢ)]. It punishes confident wrong answers very heavily.
The gradient of the loss is the list of its partial derivatives, one per parameter. Each says how the loss changes when that parameter increases a little. For the line ŷ = w x + b with MSE:
- ∂L/∂w = (2/n) Σ (ŷᵢ − yᵢ) xᵢ
- ∂L/∂b = (2/n) Σ (ŷᵢ − yᵢ)
Gradient descent moves every parameter a small step against its derivative, because the gradient points uphill:
w ← w − η ∂L/∂w, b ← b − η ∂L/∂b
where η is the learning rate. Repeat until the loss stops falling. If η is too small, training takes forever; if it is too large, each step overshoots the minimum and the loss can explode.
To test a gradient formula, compare it with a numerical gradient: (L(θ + h) − L(θ − h)) / (2h) for a small h. If they disagree, the formula or the code is wrong. This gradient check is how deep learning frameworks are tested.
Syntax
for step in range(steps):
dw, db = gradients(w, b)
w -= rate * dw
b -= rate * db
Example
xs = [1.0, 2.0, 3.0, 4.0]
ys = [3.0, 5.0, 7.0, 9.0]
def loss(w, b):
return sum((w * x + b - y) ** 2 for x, y in zip(xs, ys)) / len(xs)
def gradients(w, b):
n = len(xs)
dw = sum(2 * (w * x + b - y) * x for x, y in zip(xs, ys)) / n
db = sum(2 * (w * x + b - y) for x, y in zip(xs, ys)) / n
return dw, db
for rate in [0.05, 0.2]:
w, b = 0.0, 0.0
print(f"learning rate {rate}:")
for step in range(1, 201):
dw, db = gradients(w, b)
w, b = w - rate * dw, b - rate * db
if step in (1, 10, 50, 200):
print(f" step {step:3}: w = {w:.4g}, b = {b:.4g}, loss = {loss(w, b):.4g}")Line by line
- The data lie exactly on y = 2x + 1, so the best parameters are w = 2 and b = 1.
gradientsuses the formulas above;lossis the mean squared error.- With learning rate 0.05, the loss falls steadily and w and b approach 2 and 1.
- With learning rate 0.2, every step overshoots, and the parameters swing further each time.
- After 200 steps the large learning rate has made the loss astronomically large: training has diverged.
Expected result
learning rate 0.05:
step 1: w = 1.75, b = 0.6, loss = 1.129
step 10: w = 2.084, b = 0.7535, loss = 0.01015
step 50: w = 2.046, b = 0.8652, loss = 0.003036
step 200: w = 2.005, b = 0.986, loss = 3.29e-05
learning rate 0.2:
step 1: w = 7, b = 2.4, loss = 224.5
step 10: w = -1.033e+04, b = -3512, loss = 9.941e+08
step 50: w = -6.076e+18, b = -2.067e+18, loss = 3.439e+38
step 200: w = -1.477e+74, b = -5.024e+73, loss = 2.033e+149
In the real world
Frameworks such as PyTorch compute gradients automatically and update billions of parameters with variants of gradient descent. Choosing the learning rate is still one of the most important decisions in training; practitioners often run a quick "learning-rate finder" that tries increasing rates and watches the loss. Gradient checks are part of every framework's test suite.
Common mistakes
- Adding the gradient instead of subtracting it, which climbs the loss.
- Using MSE for probabilities when cross-entropy trains much better.
- Taking log(0) in cross-entropy; real code clips predictions slightly away from 0 and 1.
- Changing the learning rate and the model at the same time, so it is unclear which helped.
Best practices
- Print the loss regularly; a loss that goes up means the learning rate is too high or the gradient is wrong.
- Check hand-written gradients against finite differences on a tiny example.
- Start with a moderate learning rate and adjust by factors of 3 or 10.
Summary
The loss turns "how wrong" into one number: MSE for values, cross-entropy for probabilities. The gradient gives each parameter's effect on the loss, and gradient descent steps every parameter against it, scaled by the learning rate. Too large a rate diverges; a finite-difference check verifies gradients.
Quick check
Key takeaways
- Training is minimizing a loss: step each parameter by −learning rate × its partial derivative.
- Too small a learning rate crawls; too large a learning rate overshoots and can diverge.
- A finite-difference check catches bugs in hand-written gradients.
Practice problems for this lesson
My notes
Write down what you want to remember from this lesson. Notes are saved on this device as you type, and appear on your dashboard.