Decision Trees

Learn a decision tree as a sequence of questions that progressively removes uncertainty.

A university-style path from hand-calculated entropy to mixed-data classification, recursive fitting, pruning, regression trees, evaluation, and practical model choice.

The central learning question

Opening question

If several questions can divide the same training data, how should a tree decide which question to ask first?

A tree tries each allowed question and measures how much cleaner the labels become. For classification, entropy or Gini measures the mixture. Information gain measures the reduction.

\[ \text{best question}=\arg\max_{\text{question}}\left(\text{impurity before}-\text{impurity after}\right) \]

Intuition: ask the question that removes the most uncertainty now, then repeat the same idea inside each resulting group.

feature-space cuts Feature 2 ≤ 4.5? Feature 1 ≤ 2? Class 0 Class 1 Class 0

Every root-to-leaf path corresponds to one region of feature space.

Three connected examples

Play cricket?

A tiny categorical dataset for complete entropy and information-gain calculations.

Approve a loan?

A mixed numerical and categorical dataset for thresholds, recursion, prediction, and fairness.

Predict house price

A regression example for means, squared error, piecewise predictions, and extrapolation.

Session story

Ask a questionMeasure impurityChoose the best splitRepeat recursivelyControl complexity

The mathematical expression on every page is followed by its symbols, numerical substitution, plain-language intuition, and a visual consequence.

Roadmap

1

Intuition and anatomy

Questions, branches, leaves, regions, classification, and regression.
2

Entropy and Gini

Uncertainty, surprise, impurity curves, and complete calculations.
3

Information gain

Weighted child entropy and the complete cricket root split.
4

Mixed-data splits

Loan data, numeric thresholds, categories, and tie-breaking.
5

Recursion

Tree growth, stopping cases, prediction paths, and probabilities.
6

Pruning

Underfitting, overfitting, pre-pruning, and cost complexity.
7

Regression trees

SSE reduction, leaf means, step functions, and limitations.
8

Evaluation

Metrics, cross-validation, tuning, imbalance, and calibration.
9

Practice

Preprocessing, missing values, interpretation, and workflows.
10

Model choice

When trees work, when they struggle, nuances, and ensembles.
Next: Intuition