Part 10
Use a tree when threshold rules and interactions match the problem—and its instability is acceptable.
Model choice depends on data geometry, sample size, error costs, explanation needs, extrapolation, probability quality, and operational constraints.
Strong reasons to try a tree
Model-choice question
Does the problem naturally sound like “if this, and then that”?
Threshold effects
Risk changes after meaningful cutoffs rather than smoothly and linearly.
Feature interactions
One feature matters differently depending on another feature's value.
Limited preprocessing
Numeric scaling is unnecessary and nonlinear relationships emerge automatically.
Compact rule explanation
A shallow validated tree can communicate a prediction path clearly.
Mixed operational data
After suitable categorical handling, trees work naturally with tabular features.
Fast prediction
A fitted tree evaluates only a short sequence of node questions.
Situations where one tree struggles
| Situation | Why a single tree struggles | Consider |
|---|---|---|
| Smooth additive relationship | Stepwise regions approximate a smooth surface inefficiently. | Linear, generalized additive, spline, or kernel models. |
| Need extrapolation | Regression trees repeat existing leaf means outside training ranges. | Models with a defensible functional trend. |
| Small noisy dataset | Early splits can change sharply with a few rows. | Regularized linear models, resampling, or ensembles. |
| High-dimensional sparse text | Many binary split candidates and unstable local structure. | Naive Bayes or linear models as strong baselines. |
| Stable probabilities required | Leaf frequencies are coarse and can be extreme. | Calibration or probabilistic models. |
| Rotated linear boundary | Axis-aligned cuts may need a staircase of many leaves. | Logistic regression or SVM. |
Practical application patterns
| Application | Why trees may help | Nuance |
|---|---|---|
| Customer churn | Thresholds and interactions among tenure, usage, and support. | Class imbalance, intervention cost, and temporal leakage. |
| Credit operations | Rules are inspectable and nonlinear interactions are common. | Regulation, fairness, proxy variables, calibration, and appeal rights. |
| Fraud triage | Local threshold combinations can flag unusual behavior. | Extreme imbalance, adversarial adaptation, and precision workload. |
| Medical risk support | Interactions can be displayed as simple pathways. | Clinical validation, missingness, subgroup safety, and no autonomous diagnosis. |
| House pricing | Captures neighborhoods, thresholds, and interactions. | Poor extrapolation and potential spatial leakage. |
| Quality control | Sensor thresholds can map naturally to operational rules. | Drift, measurement error, and asymmetric failure costs. |
Compare with familiar models
| Property | Decision tree | Linear/logistic regression | KNN | SVM |
|---|---|---|---|---|
| Learned shape | Axis-aligned regions | Global linear effect unless transformed | Local neighborhoods | Maximum-margin boundary; kernels can curve |
| Scaling | Usually unnecessary | Useful for regularization/optimization | Essential | Usually essential |
| Interactions | Automatic through paths | Must be added explicitly | Implicit locally | Possible through kernels/features |
| Extrapolation | Poor for regression | Continues fitted trend | Predicts from observed neighbors | Depends on formulation/kernel |
| Interpretation | Rules are direct when shallow | Coefficients describe global effects | Explain by neighbors | Margin/support vectors; kernels are less direct |
The important nuances
- A tree is nonlinear overall but each numeric node is an axis-aligned one-feature split.
- High training accuracy can be evidence of memorization, not quality.
- A readable tree is not automatically fair, causal, stable, or correct.
- Impurity reduction and evaluation metrics solve different problems.
- Different but nearly equivalent trees can arise from ties or small sample changes.
- Categorical encoding determines which questions the implementation can ask.
- Regression leaves return local constants and therefore do not extrapolate smooth trends.
- Class weighting changes split calculations, not merely the final threshold.
- Feature importance measures use inside one fitted model, not the effect of an intervention.
Why ensembles usually predict better
Ensemble question
If one tree has high variance, what happens when many diverse trees vote or average?
| Method | Core idea | Connection to one tree |
|---|---|---|
| Bagging | Fit trees on bootstrap samples and average. | Reduces variance. |
| Random forest | Bagging plus random feature subsets at each split. | Decorrelates trees to improve averaging. |
| Gradient boosting | Add shallow trees sequentially to repair current errors. | Uses trees as flexible weak learners. |
A single small tree remains valuable for learning, auditing, and baseline explanations. Ensembles often improve accuracy at the cost of direct visual simplicity.
Final decision checklist
- Do threshold rules and interactions plausibly describe the problem?
- Is a piecewise-constant regression output acceptable?
- Can categories and missingness be handled without leakage?
- Are leaf probabilities supported and calibrated well enough?
- Does validation show stable performance across folds and groups?
- Can the organization accept tree instability after retraining?
- Have fairness, error cost, drift, and monitoring been addressed?
- Does a simpler baseline or an ensemble perform materially better?
The complete mental model
\[
\boxed{\text{Choose the question that most improves local homogeneity, then repeat.}}
\]
Intuition: classification makes labels more alike; regression makes target numbers closer together. Regularization stops the repeated questions before they become memorized exceptions.