Bagging means bootstrap aggregating: train separately, then combine.
Bagging is a general ensemble recipe. Random Forest uses this recipe with decision trees and adds random feature selection.
The bagging recipe
Sequence question
Which should happen first: aggregate predictions from one tree, or train many trees on different bootstrap samples?
- Draw a bootstrap sample of rows.
- Fit a base learner using that sample.
- Repeat for \(B\) learners independently.
- Aggregate their predictions.
Intuition: every learner is a slightly different answer to the same problem; the final answer is their consensus.
Worked classification vote
Vote before calculation
Three trees predict Approved, Approved, and Declined. What does hard voting return?
| Tree | Prediction | Leaf probability of Approved |
|---|---|---|
| 1 | Approved | 4/5 = 0.80 |
| 2 | Approved | 3/4 = 0.75 |
| 3 | Declined | 1/4 = 0.25 |
Intuition: hard voting keeps only each tree's final label. Soft voting preserves how confident the trees were. Scikit-learn's RandomForestClassifier averages the trees' class probabilities.
Worked regression average
Average question
Five trees predict 48, 55, 72, 50, and 65. What single number should bagging report?
The 72 is not allowed to dominate the answer by itself. It contributes one vote among five numeric opinions.
Individual trees can be jagged and variable. Their average is usually more stable, though it is still made from tree-shaped regions.
Bagging can reduce variance, but averaging cannot repair a shared systematic mistake.