Training, testing and overfitting

A model that is great on the examples it learned from may be useless on new ones. A student who memorizes last year's answers scores 100% on last year's paper and fails this year's. So we never judge a model by its training data.

Train and test

Split the data before training:

• the training set (say 80%): the model learns from it;
• the test set (20%): kept hidden, used once at the end to estimate how well the model does on new data.

Shuffle before splitting (with a fixed random seed, so the split can be repeated), or the test set may hold only the last, unusual examples.

Overfitting

# Hours studied, and whether each student passed. Two are unusual: 2 hours and passed, 7 hours and failed.
train_hours = [1, 2, 3, 4, 5, 6, 7, 8]
train_passed = [0, 1, 0, 1, 1, 1, 0, 1]
test_hours = [1.5, 2.5, 6.5, 7.5]
test_passed = [0, 0, 1, 1]

def memorizer(h):
    """The label of the most similar training student."""
    best = 0
    for i in range(len(train_hours)):
        if abs(train_hours[i] - h) < abs(train_hours[best] - h):
            best = i
    return train_passed[best]

def simple_rule(h):
    return 1 if h >= 4 else 0

def accuracy(model, hours, passed):
    right = 0
    for h, p in zip(hours, passed):
        right += model(h) == p
    return right / len(hours)

for name, model in [("memorizer", memorizer), ("simple rule", simple_rule)]:
    print(f"{name:12} train {accuracy(model, train_hours, train_passed):.0%}, test {accuracy(model, test_hours, test_passed):.0%}")

Two models learn from the same 8 students, two of whom are unusual (one passed after 2 hours, one failed after 7):

• The memorizer answers with the label of the most similar training student. It gets every training example right, including the two odd ones, which it learns as if they were rules.
• The simple rule "pass if hours ≥ 4" gets the two odd students wrong.

On new students, the simple rule wins. The memorizer learned the noise along with the pattern: it overfits.

Regularization adds a penalty for large weights to the loss, so the model prefers simple explanations.

Bias and variance

• Bias is error from assumptions that are too simple (a straight line through a curve): the model is wrong in the same way on any training set.
• Variance is error from being too sensitive to the training set: retrain it on slightly different data, and it changes a lot.

Simple models have high bias and low variance; complex ones the reverse. The bias-variance trade-off is choosing the complexity between them.

Validation and cross-validation

Choosing between models (or settings such as k in k-NN) by their test score quietly trains on the test set. So split three ways: train (learn), validation (choose), test (report once).

With little data, k-fold cross-validation does better: split the data into k parts (say 5); train on 4 and validate on the fifth, 5 times, each part taking its turn; average the 5 scores.

Mark lesson complete