Introduction
A model that overfits usually does it with large weights: it twists itself to pass through every noisy training point. Regularization fixes this by charging the model for its weights, so it only uses a large weight when the data really demands it. It is the "regularization" cure from the table in the lesson on overfitting, and almost every real model uses some form of it.
How it works
Add a penalty to the loss:
ridge (L2): loss = Σ (ŷᵢ − yᵢ)² + λ × Σ wⱼ²
lasso (L1): loss = Σ (ŷᵢ − yᵢ)² + λ × Σ |wⱼ|
λ (lambda) sets the strength. λ = 0 is plain least squares. As λ grows, the weights shrink towards 0, the model becomes simpler, and its training error rises a little, while its error on new data often falls. Too much, and the model flattens into a horizontal line that ignores the feature: underfitting. The bias b is normally not penalized, so the line can still move up and down.
For one feature, ridge regression has a formula, almost the least-squares one:
w = Σ (xᵢ − x̄)(yᵢ − ȳ) / (Σ (xᵢ − x̄)² + λ)
b = ȳ − w × x̄
The λ in the bottom of the fraction is what shrinks w. The two penalties behave differently: L2 shrinks every weight smoothly, while L1 pushes some weights all the way to exactly 0, which also selects features.
λ is a hyperparameter: it is not learned from the training data (that would always choose λ = 0, the best fit to the training points). Instead, train with several values of λ and keep the one with the smallest validation error.
With gradient descent, the L2 penalty adds 2λw to each weight's gradient, so every step also shrinks the weight a little: weight decay. Another popular regularizer needs no penalty at all: early stopping watches the validation loss during training and keeps the weights from the epoch where it was lowest.
Example
# Four noisy training points (the true pattern is y = x) and three validation points.
train = [(1, 1.0), (2, 2.6), (3, 2.4), (4, 4.9)]
valid = [(0, 0), (5, 5), (6, 6)]
n = len(train)
x_mean = sum(x for x, _ in train) / n
y_mean = sum(y for _, y in train) / n
sxy = sum((x - x_mean) * (y - y_mean) for x, y in train)
sxx = sum((x - x_mean) ** 2 for x, _ in train)
def mse(w, b, points):
return sum((w * x + b - y) ** 2 for x, y in points) / len(points)
for lam in [0, 1, 10]:
w = sxy / (sxx + lam) # ridge: the penalty λw² pulls the weight towards 0
b = y_mean - w * x_mean
print(f"lambda {lam:2}: w = {w:.3f}, train MSE {mse(w, b, train):.3f}, validation MSE {mse(w, b, valid):.3f}")Line by line
- The four training points follow y = x with noise; the third point is low and the fourth high, which makes the plain slope too steep.
sxyandsxxare the sums in the least-squares formula, computed once.- For each λ, the ridge formula divides by
sxx + lam, so a larger λ gives a smaller weight. msemeasures the line on the training points and on the validation points, which the model never saw.- λ = 0 fits the training points best but misses the validation points; λ = 1 is worse on training and far better on validation; λ = 10 is too strong and underfits both.
Expected result
lambda 0: w = 1.150, train MSE 0.304, validation MSE 0.315
lambda 1: w = 0.958, train MSE 0.350, validation MSE 0.043
lambda 10: w = 0.383, train MSE 1.038, validation MSE 2.864
In the real world
Ridge and lasso are standard in statistics and finance, where there are many correlated features and few examples. Neural networks are trained with weight decay and early stopping almost by default, and deep-learning libraries expose weight decay as one setting of the optimizer. Lasso's habit of zeroing weights is used to find the handful of genes or sensors that matter among thousands.
Common mistakes
- Choosing λ on the training data, which always picks λ = 0.
- Penalizing the bias, which drags the whole line towards zero for no reason.
- Using a penalty on unscaled features: a feature measured in large units needs a small weight and is barely penalized, so scale first.
- Expecting regularization to fix bad data: it reduces variance, not mistakes in the labels.
Best practices
- Try λ values spread over orders of magnitude, such as 0.01, 0.1, 1, 10 and 100.
- Pick λ on a validation set or with cross-validation, then report the test score once.
- Use L1 when you suspect only a few features matter, L2 otherwise.
Summary
Regularization adds a penalty for large weights to the loss: λΣw² for ridge, λΣ|w| for lasso. The penalty shrinks the weights, trading a little training error for better results on new data, and λ is chosen on validation data. Weight decay is the L2 penalty inside gradient descent, and early stopping keeps the weights from the best validation epoch.
Quick check
Key takeaways
- Regularization trades a little training error for a smaller test error.
- λ = 0 is no penalty; a huge λ flattens the model into underfitting; validation picks the value between.
- The bias (intercept) is usually not penalized.
Practice problems for this lesson
My notes
Write down what you want to remember from this lesson. Notes are saved on this device as you type, and appear on your dashboard.