Regression and optimization
Fitting lines with formulas and with gradient descent, and keeping models simple with regularization.
About 5 hours of study
Learning objectives
- Fit and judge a least-squares line
- Train with gradient descent
- Regularize with ridge, lasso and early stopping
Lessons
- 1
Linear regression Intermediate
Predict a number with a straight line: residuals and the mean squared error, the least-squares formulas, the R² score, several features, and reading a model without confusing correlation with causation.
- 2
Gradient descent Intermediate
How models learn when there is no formula: follow the slope of the loss downhill. The update rule, the learning rate, batch, stochastic and mini-batch steps, and why the loss curve flattens.
- 3
Regularization: ridge, lasso and early stopping Intermediate
Fight overfitting by penalizing large weights: the L2 (ridge) and L1 (lasso) penalties, choosing their strength on validation data, weight decay in gradient descent and early stopping.
Module practice
- Predictions and error of a lineEasy
- The least-squares lineMedium
- Linear regression with two featuresHard
- Gradient descent on one numberEasy
- Gradient descent stepsMedium
- Stochastic gradient descent for a lineHard
- L1 and L2 weight penaltiesEasy
- A ridge regression lineMedium
- Choose λ with a validation setHard