Bagging, also known as bootstrap aggregation, is a machine learning ensemble technique that combines multiple learners in order to reduce variance and improve generalization. It is generally implemented to improve the performance of supervised learning techniques. In bagging, subsamples of the entire dataset are created, with replacement, and each model is trained on a different subsample. The output of the multiple models are then combined together to form a final prediction.
The idea behind bagging is to reduce the variance of the prediction model. Variance refers to the degree to which the prediction model’s output will differ, given different datasets. High variance can lead to overfitting, which means the model is overly sensitive to small changes in the input dataset. Bagging works to reduce variance by taking multiple samples from the data and training multiple models on them. The models take into account the inherent uncertainty in predictions, and the data points that have greater influence are averaged out across all the models. This reduces variance and prevents problems due to non-generalizable features.
Bagging is generally used in combination with decision tree-based algorithms, and it can also be applied to other forms of machine learning. It is most effective when there are a large number of training examples and when only a few of them are necessary for effective decision making. Bagging can also be used to reduce the effects of overfitting, by reducing the number of parameters in an algorithm.
Bagging has proven to be a very powerful technique to improve the accuracy and generalization performance of machine learning models. While it is not always necessary, it is a cost-effective way to get the most out of a dataset.
Sources:
Birhanu, N.; Doshi, V.; Mishra, B. (2017). An Introduction to Ensemble Techniques. Journal of Big Data, 4(4). doi:10.1186/s40537-017-0075-z.
Suganthi, R.; Krishnaveni, G. (2016). Bagging: A powerful Ensemble Method for Improving the Accuracy of Classification Algorithms. International Journal of Computer Applications, 135(4). Retrieved from https://pdfs.semanticscholar.org/e7de/ebda0f0d6f145d82a7e8cef6864b1e00e2b4.pdf.
