Probabilities, trees and ensembles
Bayes' rule and Naive Bayes, decision trees, and combining many models into one.
About 5 hours of study
Learning objectives
- Apply Bayes' rule and Naive Bayes
- Grow decision trees with information gain
- Explain bagging, random forests and boosting
Lessons
- 1
Bayes' rule and Naive Bayes Intermediate
Update a belief with evidence using Bayes' rule, avoid the base-rate fallacy, and build a Naive Bayes spam filter with word counts, Laplace smoothing and log probabilities.
- 2
Decision trees Intermediate
Classify by asking one question after another: entropy, information gain and the Gini index, thresholds for numeric features, and how depth limits and pruning stop a tree from memorizing.
- 3
Ensembles: random forests and boosting Intermediate
Many weak models can beat one strong one: majority voting, bagging with bootstrap samples, random forests, and boosting, where each new model fixes the mistakes of the last.
Module practice
- Bayes' rule for a medical testEasy
- Smoothed word probabilitiesMedium
- A Naive Bayes spam filterHard
- Gini impurity of a groupEasy
- Information gainMedium
- Find the best splitHard
- Majority vote of classifiersEasy
- A bootstrap sample and its out-of-bag examplesMedium
- One round of AdaBoostHard