Now write the complete Gradient Boosting procedure.
The algorithm repeats one disciplined loop: inspect the loss at the current model, turn that information into correction targets, fit a small tree, choose leaf corrections, and update cautiously.
Initialization question
Before fitting any tree, what prediction should minimize the chosen loss?
Step 0: choose the best constant
The initial model ignores features but respects the loss. For squared error, \(F_0\) is the mean target. For absolute error, it is a median. For binary log loss, it corresponds to the class prior on the probability scale.
Why start here? The trees should learn feature-dependent structure, rather than spend their first round rediscovering the best global constant.
Round question
At boosting round \(m\), what exactly becomes the training target for the new tree?
Steps 1–4: one boosting round
- Calculate pseudo-residuals.\[r_{im}=-\left.\frac{\partial L(y_i,F(x_i))}{\partial F(x_i)}\right|_{F=F_{m-1}}\]Each row now has a desired direction of change.
- Fit a regression tree to \((x_i,r_{im})\). The tree divides feature space into leaves \(R_{1m},\ldots,R_{Jm}\), grouping rows that need similar corrections.
- Choose the best value for each leaf.\[\gamma_{jm}=\arg\min_{\gamma}\sum_{x_i\in R_{jm}}L\!\left(y_i,F_{m-1}(x_i)+\gamma\right)\]For squared error, this is the mean residual inside the leaf. For other losses, it may be another loss-specific value.
- Add a shrunken correction.\[F_m(x)=F_{m-1}(x)+\eta\sum_{j=1}^{J}\gamma_{jm}\mathbf{1}(x\in R_{jm})\]The indicator chooses the leaf reached by \(x\). The learning rate \(\eta\) controls the size of the update.
Repeat these steps for \(m=1,2,\ldots,M\). The final model is \(F_M\).
Learning-rate question
Why take only 10% of a useful correction instead of adding the whole tree?
Learning rate and number of trees work together
A small learning rate makes each round cautious. The ensemble usually needs more trees, but the finer steps can improve generalization. A large learning rate moves faster and can overshoot useful structure or fit noise sooner.
Illustrative behavior. Training loss tends to decrease as trees are added; validation loss determines when further correction stops helping new data.
Practical consequence: choose the number of rounds with validation or cross-validation. The smallest training loss is not the model-selection target.
Tree-size question
What can a depth-1 tree correct that a depth-4 tree can correct, and what can it not?
Tree complexity controls the shape of each correction
| Control | What changes | Teaching intuition |
|---|---|---|
| Tree depth / leaf count | Complexity of one correction | Shallow trees learn broad patterns; deeper trees can express higher-order interactions. |
| Learning rate \(\eta\) | Amount added from each tree | How cautious each correction is. |
| Number of trees \(M\) | Number of correction opportunities | How long the model keeps improving the training objective. |
| Minimum samples per leaf | Support required for a local correction | Prevents tiny groups from receiving very specific updates. |
| Subsample fraction | Rows used at each round | Adds randomness and can reduce variance. |
Boosting can chase outliers and mislabeled rows because they continue to produce large loss or gradients. Robust losses, conservative trees, shrinkage, validation, and data review all matter.
Common misunderstandings
| Misunderstanding | Correction |
|---|---|
| Every tree predicts the original target. | Later trees learn correction signals created by the current model and chosen loss. |
| Gradient Boosting always fits ordinary residuals. | That is true for squared loss; generally it fits negative gradients. |
| The trees are averaged like Random Forest. | Their contributions are added sequentially to the current prediction. |
| A lower training loss means a better model. | Generalization must be assessed on validation data that represents deployment. |
| More trees can only help. | Training loss can keep falling after validation performance starts worsening. |
Final check
- Why does tree 2 receive a different target from tree 1?
- For squared error, why is the correction target \(y-F\)?
- Why does Gradient Boosting fit regression trees even when the eventual task is classification?
- What do \(\eta\), tree depth, and \(M\) each control?
- Why can the algorithm predict corrections for unseen rows?
One complete sentence: Gradient Boosting builds an additive prediction function by repeatedly fitting a small regression tree to the negative gradient of the current loss and adding a shrunken leaf-wise correction.