Advanced level · Module 1 of 6 · 22 guided hours
Ensemble Methods
Combine many models with bagging, boosting and stacking for strong performance, and interpret what they learn.
Learning objectives
- Train and tune random forests and gradient boosting models
- Combine models with stacking
- Interpret ensembles with permutation importance and partial dependence
Topics
- bagging and random forests
- gradient boosting
- XGBoost, LightGBM and HistGradientBoosting
- early stopping
- stacking
- permutation importance and partial dependence
Lessons
-
Lesson 1 Bagging and Random Forests
Reduce variance by averaging many decision trees trained on different samples.
bootstrap samplebaggingrandom forestout-of-bag score
-
Lesson 2 Gradient Boosting
Build strong models by adding trees that correct previous errors, and control overfitting with learning rate and early stopping.
boostinglearning rateearly stoppingHistGradientBoosting / XGBoost / LightGBM
-
Lesson 3 Stacking and Model Combination
Combine different model types with stacking and decide when the extra complexity is worth it.
stackingmeta-modelout-of-fold predictionscomplexity cost
-
Lesson 4 Interpreting Ensembles
Find out which features drive an ensemble's predictions and how, using model-agnostic methods.
permutation importanceimpurity importance pitfallspartial dependencecorrelated features
Practical activity
Predict household food insecurity in a provided synthetic survey data set with random forests and gradient boosting, tune them with early stopping and explain the most influential features.
Expected outcomes
- Strong, well-tuned tabular models
- Honest interpretation of ensembles
Assessment
Practical: beat a provided benchmark on a held-out set and explain the model with two interpretation methods.
Taught live by an instructor — online or in person. Free course lessons are for self-study and do not lead to a certificate — see registration options.