Bagging and Boosting in Ensemble Learning of ML

Bagging and boosting are both ensemble learning methods, which means they combine multiple models to create a more accurate and robust model than any single model could be.

Bagging (short for bootstrap aggregating) works by creating multiple copies of the training dataset, each of which is created by sampling with a replacement from the original dataset. Each of these copies is then used to train a separate model, such as a decision tree or a linear regression model. The predictions of the individual models are then combined to create a final prediction.

Bagging is effective at reducing the variance of a model, which is the tendency for a model to overfit the training data. This is because each of the individual models in the ensemble is trained on a different subset of the data, which helps to prevent them from all overfitting the same way.

Boosting also works by creating multiple models, but it does so in a sequential manner. In the first iteration, a model is trained on the entire training dataset. In the next iteration, the model is trained on the training data, but the weights of the data points are adjusted so that the model pays more attention to the data points that were misclassified in the previous iteration. This process is repeated until a desired number of models have been created.

Boosting is effective at reducing the bias of a model, which is the tendency for a model to underfit the training data. This is because the models in the ensemble are trained to correct the mistakes of the previous models.

Here is an example of how bagging and boosting can be used to improve the accuracy of a model. Let's say we have a dataset of 1000 data points, and we want to build a model to predict whether a customer will churn (cancel their subscription). We could build a single decision tree model on the entire dataset, but this model might overfit the training data and not generalize well to new data.

Instead, we could use bagging to create 100 decision trees, each of which is trained on a different bootstrap sample of the original dataset. The predictions of the 100 decision trees can then be combined to create a final prediction. This approach is likely to produce a more accurate model than a single decision tree, because the bagging technique will help to reduce the variance of the model.

We could also use boosting to improve the accuracy of our model. In this case, we would start by training a simple decision tree on the entire dataset. In the next iteration, we would train a second decision tree on the training data, but the weights of the data points would be adjusted so that the model pays more attention to the data points that were misclassified by the first decision tree. This process would be repeated until a desired number of decision trees had been created. The predictions of the decision trees would then be combined to create a final prediction.

Boosting is likely to produce a more accurate model than bagging in this case, because it is specifically designed to reduce the bias of a model. However, bagging is typically easier to implement and less computationally expensive than boosting.

In general, bagging is a good choice when the goal is to reduce the variance of a model, while boosting is a good choice when the goal is to reduce the bias of a model. The best approach to use will depend on the specific problem being solved.

Photo by Elif Dörtdoğan and Jonas Svidras