What if the next tree could study the mistakes of the trees before it?
Bagging taught us that many trees can be better than one. Boosting keeps the ensemble idea but changes how its members learn: models arrive one after another, and every new learner is chosen to improve the current combined model.
Recall before reveal
In Random Forest, does tree 20 know what trees 1–19 predicted incorrectly?
Where we left off: bagging
In bagging, every tree receives its own bootstrap sample and solves the same original prediction problem. Random Forest also samples candidate features at each split. These choices make the trees different enough that averaging can cancel part of their individual variance.
Tree \(T_b\) does not wait for \(T_{b-1}\). The trees can be trained independently, then averaged for regression or combined through averaged class probabilities for classification.
The transition: bagging reduces instability by averaging independent learners. Boosting tries to reduce the current model's loss by adding dependent learners.
Start with a limitation
A shallow decision tree cannot represent a complicated relationship. Is that always a weakness?
Why use weak learners?
A weak learner is deliberately simple. In tree boosting it is usually a shallow regression tree, often with only a few leaves. One such tree has limited ability, but it is fast, regularized by its simplicity, and easy to use as one correction inside a larger model.
“Weak” does not mean useless or necessarily barely better than random in every modern implementation. It means the base learner has restricted capacity compared with the final ensemble.
One deep tree
Attempts to learn the whole relationship at once. It may fit detailed noise and change sharply with the sample.
Many small trees
Each tree performs a limited correction. Their sum can represent a rich nonlinear function.
Additive model: the final prediction is the starting prediction plus contributions from many trees. We do not choose one winning tree.
Sequence question
If tree 2 is supposed to fix tree 1's remaining mistakes, can the two trees be trained independently?
The defining property: later learners depend on earlier learners
At round \(m\), the ensemble already has prediction function \(F_{m-1}(x)\). We calculate a correction target from its current errors, fit a small tree \(h_m(x)\) to that target, and add the tree:
This dependence creates the learning sequence. Tree 2 is solving a different problem from tree 1 because the current predictions have changed. Tree 3 sees what remains after both earlier contributions.
Each model includes all trees added up to that point. The next correction is calculated from the updated model.
Comparison question
Random Forest also combines many trees. Why is it not boosting?
Bagging and boosting combine trees differently
| Bagging / Random Forest | Boosting | |
|---|---|---|
| Training relationship | Trees can be trained independently. | Each tree depends on the current ensemble. |
| How diversity is created | Different sampled rows; Random Forest also samples features. | Each stage receives a new correction problem. |
| Combination | Average predictions or class probabilities. | Add successive tree contributions. |
| Main behavior | Averaging reduces the instability of individual trees. | Sequential additions reduce the chosen training loss. |
| Parallel training | Individual trees are naturally parallel. | Rounds are sequential because round \(m\) needs \(F_{m-1}\). |
Picture the difference: bagging asks many people to solve the original problem independently and averages their answers. Boosting asks one person to draft, the next to correct the current draft, and another to correct what still remains.
The key missing piece
How should we define the “error” that the next tree must fix?
For squared-error regression, the answer looks familiar: fit the next tree to residuals. Gradient Boosting generalizes that answer by using the negative gradient of a loss function. We will discover this first through numbers.