Accuracy, the share of predictions that are right, is the obvious score. It can also be badly misleading.
The confusion matrix
For a yes/no classifier (positive = the thing we look for: spam, a disease, fraud), every prediction lands in one of four boxes:
| Actual class | Predicted positive | Predicted negative |
|---|---|---|
| Actually positive | TP (true positive) | FN (false negative: missed) |
| Actually negative | FP (false positive: false alarm) | TN (true negative) |
From them:
| Score | Formula | Answers |
|---|---|---|
| accuracy | (TP + TN) / all | how often is it right? |
| precision | TP / (TP + FP) | when it says yes, how often is it right? |
| recall (sensitivity) | TP / (TP + FN) | of the real yes cases, how many does it find? |
| F1 score | 2 × precision × recall / (precision + recall) | one number that is high only if both are |
def scores(actual, predicted):
tp = sum(a == 1 and p == 1 for a, p in zip(actual, predicted))
fp = sum(a == 0 and p == 1 for a, p in zip(actual, predicted))
fn = sum(a == 1 and p == 0 for a, p in zip(actual, predicted))
tn = sum(a == 0 and p == 0 for a, p in zip(actual, predicted))
accuracy = (tp + tn) / len(actual)
precision = tp / (tp + fp) if tp + fp else 0.0
recall = tp / (tp + fn) if tp + fn else 0.0
f1 = 2 * precision * recall / (precision + recall) if precision + recall else 0.0
return (tp, fp, fn, tn), round(accuracy, 2), round(precision, 2), round(recall, 2), round(f1, 2)
actual = [1, 1, 1, 1, 0, 0, 0, 0, 0, 0]
predicted = [1, 1, 1, 0, 1, 0, 0, 0, 0, 0]
print("spam filter (TP, FP, FN, TN), accuracy, precision, recall, F1:")
print(scores(actual, predicted))
fraud = [1, 1] + [0] * 18
always_no = [0] * 20
print("always 'no' on 20 payments, 2 of them fraud:")
print(scores(fraud, always_no))The accuracy trap
In the second part of the example, only 2 of 20 card payments are fraud. A "model" that always says no is 90% accurate, and catches no fraud at all: its recall is 0. When one class is rare (imbalanced data), accuracy says little; look at precision and recall.
Which matters more?
There is usually a trade-off: flag more cases and recall goes up while precision goes down.
- A cancer screening test must not miss sick patients: recall matters most. A false alarm only means a second test.
- A spam filter must not bin your job offer: precision matters most. A spam email in your inbox is only annoying.
- When both matter, use F1.
Quick check
Practice problems for this lesson
My notes
Write down what you want to remember from this lesson. Notes are saved on this device as you type, and appear on your dashboard.