Introduction
Deep networks have so many parameters that they can memorize almost any training set, noise and all. Such a model looks perfect on the data it has seen and disappoints on everything else. Regularization covers the techniques that keep a network from memorizing, so that it generalizes to new data. Every serious model uses several of them at once.
How it works
Split the data: train on the training set and measure on a validation set the model never trains on.
- Underfitting: both errors are high; the model is too simple or under-trained.
- Overfitting: the training error keeps falling while the validation error stops falling and starts rising.
L2 regularization adds a penalty to the loss: L_total = L_data + λ Σ w². Its gradient adds 2λw to each weight's gradient, so every update also shrinks each weight a little, which is why it is also called weight decay. Large weights are needed to fit noise sharply, so penalizing them favours smoother functions.
Dropout switches off each unit at random during training, keeping each with probability p. With inverted dropout, the kept values are divided by p, so the expected value of each unit is unchanged and nothing needs to change at inference, when dropout is turned off. In the backward pass, the gradient flows only through the kept units, scaled by the same 1/p.
Early stopping watches the validation error after each epoch, remembers the best epoch, and stops when there has been no improvement for a number of epochs called the patience. The weights from the best epoch are kept.
More data, data augmentation (such as flipped or cropped images) and smaller models also reduce overfitting.
Syntax
loss = data_loss + l2 * sum(w * w for w in weights)
w -= rate * (grad + 2 * l2 * w) # weight decay
out = [a * m / keep for a, m in zip(acts, mask)] # inverted dropout, mask of 0/1
grad_in = [g * m / keep for g, m in zip(grad_out, mask)]
Example
import random
random.seed(5)
train = [(x / 10, x / 10 * 2 + random.gauss(0, 0.3)) for x in range(-10, 11, 2)]
valid = [(x / 10 + 0.05, (x / 10 + 0.05) * 2 + random.gauss(0, 0.3)) for x in range(-10, 10, 4)]
def features(x, degree):
return [x ** k for k in range(degree + 1)]
def fit(degree, l2, epochs=600, rate=0.1):
w = [0.0] * (degree + 1)
for _ in range(epochs):
grad = [0.0] * len(w)
for x, y in train:
f = features(x, degree)
err = sum(a * b for a, b in zip(w, f)) - y
for k in range(len(w)):
grad[k] += 2 * err * f[k] / len(train)
w = [wk - rate * (g + 2 * l2 * wk) for wk, g in zip(w, grad)]
return w
def mse(w, data, degree):
return sum((sum(a * b for a, b in zip(w, features(x, degree))) - y) ** 2 for x, y in data) / len(data)
for degree, l2 in [(1, 0.0), (6, 0.0), (6, 0.01)]:
w = fit(degree, l2)
print(f"degree {degree}, L2 {l2:<4}: train error {mse(w, train, degree):.4f}, validation error {mse(w, valid, degree):.4f}")Line by line
- The data come from the line y = 2x with random noise; the validation points are new x values.
- A straight line (degree 1) has the right shape for this data.
- A degree-6 polynomial has more freedom and fits the training points more closely, noise included, so its training error is lower.
- Its validation error is higher than the straight line's: it has learned some of the noise.
- A small L2 penalty keeps the degree-6 weights small: its training error rises a little, but its validation error becomes the lowest of the three.
Expected result
degree 1, L2 0.0 : train error 0.1025, validation error 0.0491
degree 6, L2 0.0 : train error 0.0961, validation error 0.0608
degree 6, L2 0.01: train error 0.1031, validation error 0.0410
In the real world
Dropout was a key ingredient of the networks that won the ImageNet competition in 2012, and it is still used in the fully connected parts of models. Weight decay is set in every training recipe, usually through AdamW. Early stopping is built into libraries as a callback. Large language models mostly avoid overfitting by training on enormous amounts of data for about one pass.
Common mistakes
- Choosing settings by the test set, which makes the final test score optimistic; use a validation set.
- Leaving dropout on at inference, or forgetting the 1/p scaling.
- Applying weight decay to biases and normalization parameters, which usually does not help.
- Stopping at the first epoch without improvement, when the validation error is naturally noisy.
Best practices
- Always plot or print training and validation errors together.
- Start with a small amount of weight decay and dropout around 0.1–0.5 in large layers.
- Keep the best weights, not the last ones, when stopping early.
Summary
Overfitting shows as a validation error that rises while the training error falls. L2 regularization adds λ Σ w² to the loss and shrinks weights each step; dropout randomly disables units, scaling the rest by 1/p; early stopping ends training after a patience of epochs without validation improvement and keeps the best weights.
Quick check
Key takeaways
- A falling training error with a rising validation error is the signature of overfitting.
- L2 shrinks every weight a little at each step; dropout forces redundancy; early stopping limits training time.
- With inverted dropout, scale the kept values by 1/p in training so nothing changes at inference.
Practice problems for this lesson
My notes
Write down what you want to remember from this lesson. Notes are saved on this device as you type, and appear on your dashboard.