The key takeaways from this section include:
- Multiple independent estimates are consistently more accurate than any single estimate, so ensemble techniques are a powerful way for improving the quality of your models
- Sometimes you'll use model stacking or meta-ensembles where you use a combination of different types of models for your ensemble
- It's also common to have multiple similar models in an ensemble - e.g. a bunch of decision trees
- Bagging (Bootstrap AGGregation) is a technique that leverages Bootstrap Resampling and Aggregation
- Bootstrap resampling uses multiple smaller samples from the test dataset to create independent estimates, and aggregate these estimates to make predictions
- A random forest is an ensemble method for decision trees using Bagging and the Subspace Sampling method to create variance among the trees
- With a random forest, for each tree, we sample two-thirds of the training data and the remaining third is used to calculate the out-of-bag error
- In addition, the Subspace Sampling method is used to further increase variability by randomly selecting the subset of features to use as predictors for training any given tree
GridsearchCVis an exhaustive search technique for finding optimal combinations of hyperparameters- Boosting leverages an ensemble of weak learners (weak models) to create a strong combined model
- Boosting (when compared to random forests) is an iterative rather than independent process, using each iteration to strengthen the weaknesses of the previous iterations
- Two of the most common algorithms for Boosting are Adaboost (Adaptive Boosting) and Gradient Boosted Trees
- Adaboost creates new classifiers by continually influencing the distribution of the data sampled to train each successive tree
- Gradient Boosting is a more advanced boosting algorithm that makes use of Gradient Descent
- XGBoost (eXtreme Gradient Boosting) is one of the top gradient boosting algorithms currently in use
XGBoostis a stand-alone library that implements popular gradient boosting algorithms in the fastest, most performant way possible