10 Decision Trees are Better Than 1

Breaking down bagging, boosting, Random Forest, and AdaBoost

This is the 2nd article in a series on decision trees. In the last post, we introduced decision trees and discussed how to grow them using data. While they are an intuitive machine learning approach, decision trees are prone to overfitting. Here we discuss a solution to this overfitting problem via decision tree ensembles.

Key points:

• Decision tree ensembles combine several decision trees into a single estimator
• Tree ensembles are less prone to overfitting than a single decision tree

Decision trees

In the previous article of this series, I reviewed decision trees and how we can use them to make predictions. However, for many real-world problems, a single decision tree is often prone to bias and overfitting.

We saw this in our example from the last blog, where even after a little hyperparameter tuning, our decision tree was still wrong 35% of the time.