Introduction
Ask one expert and you get one opinion, mistakes included. Ask many experts who make different mistakes, and take a vote, and the mistakes tend to cancel out. Ensembles apply this idea to models: they combine many models, often simple and individually weak, into one strong predictor. For data in tables, such as sales, credit or sensor records, ensembles of decision trees are the most accurate methods in common use.
How it works
Voting. Several classifiers each predict, and the majority wins (for regression, average the predictions). If each model is right most of the time and their mistakes are independent, the majority is right much more often than any single model. If they all make the same mistakes, voting changes nothing, so the art is to make the models different.
Bagging (bootstrap aggregating) makes models different by giving each one different data. A bootstrap sample draws n examples with replacement from the n training examples: some appear twice or more, and about a third never appear. Those left out are the out-of-bag examples, a free validation set for that model. Train one model per bootstrap sample and vote.
A random forest is bagging with decision trees plus one more source of difference: at each split, a tree may only consider a random subset of the features. The trees become less alike, the vote becomes stronger, and the forest rarely overfits even with hundreds of deep trees.
Boosting works the other way round: it trains models one after another, each one focusing on the examples the earlier ones got wrong. In AdaBoost, every example has a weight. After each weak model (often a decision stump):
error ε = total weight of the misclassified examples (weights summing to 1)
α = ½ × ln((1 − ε) / ε) the model's say in the final vote
wrong examples: weight × e^α right examples: weight × e^(−α)
then divide all weights by their sum
The next model is trained on the reweighted examples, so it concentrates on the hard ones. The final prediction is a weighted vote with weights α. Gradient boosting generalizes the idea: each new tree is fitted to the remaining errors (the gradient of the loss) of the ensemble so far.
Example
# Three weak classifiers on ten examples; each makes different mistakes.
labels = [1, 0, 1, 1, 0, 1, 0, 0, 1, 1]
models = {
"A": [1, 0, 1, 0, 0, 1, 1, 0, 1, 1],
"B": [1, 1, 1, 1, 0, 0, 0, 0, 1, 0],
"C": [0, 0, 1, 1, 1, 1, 0, 0, 0, 1],
}
def accuracy(predicted):
return sum(p == y for p, y in zip(predicted, labels)) / len(labels)
for name, predicted in models.items():
print(f"model {name}: {accuracy(predicted):.0%}")
vote = [1 if sum(column) >= 2 else 0 for column in zip(*models.values())]
print("majority vote:", vote, f"{accuracy(vote):.0%}")Line by line
labelsholds the true classes of ten examples, and each model's predictions are listed under its name.accuracycounts matches, as in the lesson on measuring a classifier.- Each model alone gets 70% or 80% right.
zip(*models.values())lines up the three predictions for each example; the vote is 1 when at least two say 1.- The models' mistakes fall on different examples, so at every position at least two of the three are right.
Expected result
model A: 80%
model B: 70%
model C: 70%
majority vote: [1, 0, 1, 1, 0, 1, 0, 0, 1, 1] 100%
In the real world
Gradient-boosted trees (libraries such as XGBoost, LightGBM and CatBoost) win most machine-learning competitions on table data and run credit scoring, search ranking and demand forecasting in industry. Random forests are a reliable first choice that needs little tuning. Even large neural networks are often combined into ensembles for the last bit of accuracy, and weather forecasts are ensembles of many simulation runs.
Common mistakes
- Combining models that make the same mistakes, which only adds cost.
- Boosting for too many rounds on noisy data, which fits the noise: boosting reduces bias but can overfit.
- Forgetting the out-of-bag examples, a free estimate of a forest's accuracy.
- Losing explainability without noticing: a forest of 500 trees is not as readable as one tree.
Best practices
- Start with a random forest as a strong, low-effort baseline for table data.
- Tune boosting's number of rounds and learning rate with validation data and early stopping.
- Check that the ensemble beats its best single model on validation data.
Summary
Ensembles combine many models. Voting works when the models' mistakes differ; bagging creates differences with bootstrap samples, and random forests add random feature choices to bagged trees, reducing variance. Boosting trains models in sequence on reweighted or residual errors, reducing bias; AdaBoost gives each model a say α = ½ ln((1 − ε)/ε) and raises the weights of the examples it got wrong.
Quick check
Key takeaways
- Voting helps only when the models make different mistakes.
- Bagging and random forests reduce variance; boosting reduces bias.
- Gradient-boosted trees and random forests are the strongest methods for table-shaped data.
Practice problems for this lesson
My notes
Write down what you want to remember from this lesson. Notes are saved on this device as you type, and appear on your dashboard.