Gradient boosting is a machine learning technique for regression and classification problems, which produces a prediction model in the form of an ensemble of weak prediction models, typically decision trees. It builds the model in a stage-wise fashion like other boosting methods do, and it generalizes them by allowing optimization of an arbitrary differentiable loss function.
Gradient boosting was proposed by Friedman (2001) and has quickly become one of the most popular boosting algorithms for supervised learning. It can be used for both regression and classification tasks. For example, it can provide support for identification of customer churn (classification) and prediction of sales receipts (regression).
The workings of gradient boosting involve three elements: weak learners, an objective function to be optimized, and a search procedure. Weak learners are typically decision trees, and they work by providing simple decisions based on input variables. The objective function is a measure of the model’s predictive power, and the search procedure handles the weak learner’s selections to build the model.
At each step, a decision tree is fitted to the residuals from the model (the difference between the true values and predictions from earlier steps). With each step, the model tries to maximize the objective function through a search procedure. The weak learners are applied sequentially, with each one attempting to minimize the objective function by producing the best possible prediction.
Gradient boosting is popular because it is scalable, works on a variety of machine learning problems, and is effective. It typically produces more accurate results than other boosting methods. Additionally, it tends to work well with a variety of data types and distributions, which makes it a great choice for those just starting out with machine learning.
