One decision tree is a useful opinion. A forest makes that opinion more dependable.
A progressive path from ensemble intuition to bootstrap sampling, bagging, random feature selection, classification, regression, OOB evaluation, variance mathematics, tuning, and practical model choice.
The central question
Opening question
If a small change in the training data can change one tree, how could several trees make a more stable prediction?
Random Forest combines two sources of randomness: each tree sees a bootstrap sample of the rows, and each node considers only a random subset of features. The trees become different enough that their mistakes can cancel when their predictions are aggregated.
Intuition: one tree may be wrong for a particular row. If the other trees are not wrong in exactly the same way, averaging or voting makes the final answer steadier.
The forest aggregates many different tree opinions.
Roadmap
Why an ensemble
Single-tree instability, ensemble learning, and diversity.
Bootstrap sampling
Sampling with replacement and the 63.2% / 36.8% result.
Bagging
Bootstrap aggregating, voting, averaging, and a worked example.
Random features
Why feature subsets decorrelate trees.
Classification
A forest vote, class probabilities, and a loan example.
Regression
Averaging tree predictions and a house-price example.
Out-of-bag evaluation
Use rows excluded from each bootstrap sample.
Variance mathematics
Strength, correlation, and the ensemble variance equation.
Tuning and evaluation
Hyperparameters, metrics, imbalance, and validation.
Interpretation
MDI, permutation importance, correlation, and caution.
Practice and model choice
Workflow, limitations, and Random Forest versus boosting.
Three definitions to keep separate
| Idea | What is randomized? | What is combined? |
|---|---|---|
| Ensemble | Not necessarily randomized | Predictions from several models |
| Bagging | Bootstrap samples of rows | Predictions from base learners |
| Random Forest | Bootstrap rows plus feature subsets at each split | Predictions from decision trees |
Memory hook: ensemble is the umbrella; bagging is a way to create an ensemble; Random Forest is bagged decision trees with extra feature randomness.