Introduction
Much of data science is prediction: the price a house will sell for, how many bikes will be rented tomorrow, whether a customer will leave. Machine learning libraries make fitting models easy, but the ideas that decide whether a prediction can be trusted are simple and fit in a few lines of plain Python. This lesson builds them from scratch.
How it works
Features and target. Each row has features, the inputs, and a target, the value to predict. To predict a test score from hours studied, hours is the feature and score the target.
Training and test sets. A model that is judged on the same rows it learned from looks better than it is: it may have memorised them. So split the rows: fit the model on the training set, then measure its errors only on the test set, rows it has never seen. A common split is 80% training and 20% test.
Baselines. A model is only useful if it beats the simplest sensible guess, the baseline. For a number, the baseline predicts the training mean for every row; for a category, the most common class. If a model cannot beat that, it has learned nothing useful.
A first model: the least-squares line. With one feature, fit y = a + b·x on the training rows, with b = Σ(x − x̄)(y − ȳ) ÷ Σ(x − x̄)² and a = ȳ − b·x̄.
Another model: k-nearest neighbours. To predict for a new point, find the k training rows closest to it (by straight-line distance on the features) and take the most common label among them, or their average for a number. It needs no formula, only a sensible distance and a choice of k.
Measuring error. For each test row, the error is actual − predicted.
- MAE, the mean absolute error, is the average of the sizes of the errors: "typically off by 1.8 marks".
- RMSE, the root mean squared error, squares the errors, averages them and takes the square root. It punishes large errors more than MAE does.
Both are in the units of the target, which makes them easy to explain.
Example
# Prediction: fit on training rows, judge on test rows, beat the baseline.
train = [(1, 52), (2, 55), (3, 61), (4, 64), (5, 70), (6, 73)]
test = [(7, 77), (8, 85)]
mx = sum(x for x, _ in train) / len(train)
my = sum(y for _, y in train) / len(train)
b = sum((x - mx) * (y - my) for x, y in train) / sum((x - mx) ** 2 for x, _ in train)
a = my - b * mx
for x, y in test:
print(f"x={x}: actual {y}, line {a + b * x:.1f}, baseline {my:.1f}")
mae_line = sum(abs(y - (a + b * x)) for x, y in test) / len(test)
mae_base = sum(abs(y - my) for _, y in test) / len(test)
print(f"MAE line {mae_line:.2f}, MAE baseline {mae_base:.2f}")Line by line
- Six training rows and two test rows of (hours, score).
- The slope and intercept are computed from the training rows only.
- For each test row the program prints the actual score, the line's prediction and the baseline, the training mean.
- The last line compares the two mean absolute errors.
Expected result
x=7: actual 77, line 77.8, baseline 62.5
x=8: actual 85, line 82.2, baseline 62.5
MAE line 1.81, MAE baseline 18.50
The line is off by under 2 marks on average; the baseline by 18.5. The model has learned something real.
In the real world
Every serious modelling project reports results on held-out data and against a baseline; competitions such as Kaggle keep their test sets secret for exactly this reason. Recommendation systems use nearest-neighbour ideas ("customers like you bought…"), and forecasters report MAE or RMSE so that users know how far to trust a forecast.
Common mistakes
- Measuring error on the training data, which flatters the model.
- Skipping the baseline, so a useless model looks impressive.
- Letting test rows influence the model, for example by using their mean, called data leakage.
- Comparing MAE across targets with different units.
Best practices
- Split before you look at the test rows, and use them once at the end.
- Report a baseline next to every model.
- Report the error in the target's units, with MAE for typical size and RMSE when big errors matter.
Summary
Prediction maps features to a target. Fit on a training set, judge on a test set, and compare with a baseline such as the training mean. A least-squares line and k-nearest neighbours are two simple models, and MAE and RMSE measure their errors in the target's units.
Quick check
Key takeaways
- Fit on the training set and judge only on the test set.
- Every model must beat a simple baseline to be worth using.
- MAE is the typical error; RMSE punishes large errors more.
Practice problems for this lesson
My notes
Write down what you want to remember from this lesson. Notes are saved on this device as you type, and appear on your dashboard.