Regression and optimization

Fitting lines with formulas and with gradient descent, and keeping models simple with regularization.

About 5 hours of study

Learning objectives

Lessons

  1. 1

    Linear regression Intermediate

    Predict a number with a straight line: residuals and the mean squared error, the least-squares formulas, the R² score, several features, and reading a model without confusing correlation with causation.

  2. 2

    Gradient descent Intermediate

    How models learn when there is no formula: follow the slope of the loss downhill. The update rule, the learning rate, batch, stochastic and mini-batch steps, and why the loss curve flattens.

  3. 3

    Regularization: ridge, lasso and early stopping Intermediate

    Fight overfitting by penalizing large weights: the L2 (ridge) and L1 (lasso) penalties, choosing their strength on validation data, weight decay in gradient descent and early stopping.

Module practice

Module quiz

1. What does the mean squared error average?
2. Gradient descent moves each parameter...
3. Which penalty can set weights exactly to zero?
4. The loss grows larger every step. What should you change first?
5. A ridge weight is Sxy / (Sxx + λ). What happens as λ grows very large?
6. Why is λ not chosen by the training error?

‹ FoundationsClassification ›