Introduction
A spam filter, a fraud detector and a medical test all answer yes or no. Such a model is a classifier, and judging it by "percentage correct" alone can be badly misleading. If 1 email in 100 is spam, a filter that never flags anything is 99% accurate and completely useless. This lesson shows how to evaluate classifiers honestly.
How it works
The confusion matrix. Call the class we look for "positive" (spam = 1). Every prediction is one of four kinds:
- TP, true positive: spam, flagged as spam;
- FP, false positive: a good email flagged as spam (a false alarm);
- FN, false negative: spam that got through (a miss);
- TN, true negative: a good email let through.
These four counts form the confusion matrix, and every measure comes from them.
Accuracy = (TP + TN) ÷ all rows. It is fine when the classes are balanced, but on rare classes it rewards a model for predicting "no" every time.
Precision = TP ÷ (TP + FP): when the model says yes, how often is it right? Recall = TP ÷ (TP + FN): of all the real positives, how many did it find? When nothing is predicted positive, precision is undefined; when there are no real positives, recall is undefined.
F1 combines them: 2 × precision × recall ÷ (precision + recall), which simplifies to 2TP ÷ (2TP + FP + FN). It is high only when both are high.
Thresholds. Many models output a score, such as a probability of spam, and a threshold turns it into a decision: score ≥ 0.5 means spam. Lowering the threshold flags more emails, which raises recall and usually lowers precision; raising it does the opposite. The right threshold depends on the costs: missing a cancer is far worse than a false alarm, so medical screening favours recall; a spam filter that hides real emails annoys users, so it favours precision. Try several thresholds on test data and choose by the measure that matches the costs.
Example
# Evaluating a classifier: the confusion matrix and the measures built from it.
actual = [1, 0, 1, 1, 0, 0, 1, 0, 0, 0] # 1 = spam
predicted = [1, 0, 0, 1, 0, 1, 1, 0, 0, 0]
tp = sum(1 for a, p in zip(actual, predicted) if a == 1 and p == 1)
fp = sum(1 for a, p in zip(actual, predicted) if a == 0 and p == 1)
fn = sum(1 for a, p in zip(actual, predicted) if a == 1 and p == 0)
tn = sum(1 for a, p in zip(actual, predicted) if a == 0 and p == 0)
print("TP", tp, "FP", fp, "FN", fn, "TN", tn)
print("accuracy:", (tp + tn) / len(actual))
print("precision:", tp / (tp + fp))
print("recall:", tp / (tp + fn))Line by line
- Ten emails with their real labels and the filter's predictions; 1 means spam.
- Each count checks one combination of actual and predicted.
- Accuracy, precision and recall are computed from the four counts.
Expected result
TP 3 FP 1 FN 1 TN 5
accuracy: 0.8
precision: 0.75
recall: 0.75
Accuracy is 80%, but the precision and recall of 75% say more: one good email in four flagged was wrongly flagged, and one spam in four got through.
In the real world
Banks tune fraud models to catch most fraud while keeping false alarms low enough that customers are not blocked constantly. Medical tests are described by sensitivity (recall) and specificity. Search engines are judged by precision and recall of relevant results, and model reports in industry always show the confusion matrix next to any single score.
Common mistakes
- Reporting accuracy alone when one class is rare.
- Mixing up false positives and false negatives, which have very different costs.
- Keeping the default threshold of 0.5 without checking the trade-off.
- Choosing the threshold on the test set and then reporting the same test set as unbiased.
Best practices
- Always show the confusion matrix.
- Report precision and recall, or F1, for the class that matters.
- Choose the threshold from the real costs of each error, on data separate from the final test.
Summary
A classifier's predictions fall into TP, FP, FN and TN. Accuracy is the share correct but hides failure on rare classes; precision measures how trustworthy a yes is, recall how many positives are found, and F1 balances the two. The threshold moves the balance, and the right choice depends on what each kind of error costs.
Quick check
Key takeaways
- The confusion matrix holds all the information; every measure is built from its four counts.
- Accuracy hides failure on rare classes; report precision and recall.
- Moving the threshold trades precision against recall; choose it from the costs of each kind of error.
Practice problems for this lesson
My notes
Write down what you want to remember from this lesson. Notes are saved on this device as you type, and appear on your dashboard.