Part 9

Trees need less numerical preprocessing, but they still need careful data engineering.

A reliable tree workflow protects the validation boundary, handles categories and missingness deliberately, inspects leaf support, and treats interpretation as evidence rather than causality.

End-to-end workflow

Workflow question

If trees do not usually need scaling, does that mean preprocessing cannot leak information?

Define target and costsSplit by real deployment unitFit preprocessing on trainTune with validationTest and monitor

Define the prediction time, target window, and cost of each mistake.
Split data by person, account, location, or time as deployment requires.
Fit imputation and encoding only on training folds.
Build a simple baseline and tune tree complexity.
Evaluate metrics, calibration, fairness, stability, and latency.
Document the final rules and monitor drift after deployment.

Why scaling is usually unnecessary

\[ x_j\le t \]

If income is converted from thousands of rupees to rupees, the ordering of values is unchanged and the equivalent threshold rescales with it.

\[ Income_{thousands}\le42.5\quad\Longleftrightarrow\quad Income_{rupees}\le42{,}500 \]

Intuition: a tree cares about which rows fall on each side of a threshold, not Euclidean distance or gradient size.

Scaling can still be needed elsewhere in a pipeline, such as when a tree is compared or stacked with scale-sensitive models.

Categorical features

Strategy	Benefit	Risk
One-hot encoding	Safe for unordered categories and standard scikit-learn pipelines.	Each split isolates one encoded category; high cardinality creates many columns.
Ordinal encoding	Compact and suitable for truly ordered categories.	Invents an order when used on nominal categories.
Native categorical tree library	Can search category groupings more directly.	Behavior differs across libraries and must be validated.
Target encoding	Compact for high cardinality.	Severe leakage risk unless learned within each training fold.

Missing values

Missingness question

Can “missing income” itself carry information, and when would using that information be dangerous?

Numeric imputation can use median values plus a missingness indicator.
Categorical missingness can become an explicit `Missing` category when meaningful.
Some tree estimators and versions support missing values natively; verify the exact estimator and criterion.
Missingness created only after deployment may signal pipeline failure rather than user behavior.

Intuition: treat missingness as a data-generating event that needs investigation, not just an empty cell to fill.

A leakage-safe classification pipeline

numeric_pipe = Pipeline([ ("imputer", SimpleImputer(strategy="median")) ]) categorical_pipe = Pipeline([ ("imputer", SimpleImputer(strategy="most_frequent")), ("onehot", OneHotEncoder(handle_unknown="ignore")) ]) model = Pipeline([ ("prepare", ColumnTransformer([...])) , ("tree", DecisionTreeClassifier( criterion="entropy", random_state=42 )) ])

The complete pipeline is fitted inside each validation fold so imputers and encoders never learn from validation rows.

Interpret one tree carefully

Inspection	Question to ask
Root and early splits	Are dominant rules plausible, stable, and available at prediction time?
Leaf sample counts	Are important predictions supported by enough examples?
Class distribution	How uncertain is each leaf?
Train vs validation behavior	Do the same rules survive unseen data?
Subgroup errors	Are mistakes concentrated in protected or operationally important groups?

Feature importance is not causality

\[ Importance(j)\propto\sum_{t:\,\text{node }t\text{ splits on }j}\frac{N_t}{N}\Delta I_t \]

Intuition: a feature receives credit when the fitted tree uses it to reduce impurity for many rows.

Continuous and high-cardinality features have more chances to create an attractive split.
Correlated features can share or substitute importance.
Importance does not give direction: it does not say whether higher values raise or lower predictions.
Predictive association does not prove that changing the feature changes the outcome.

Compare impurity importance with validation-set permutation importance and domain knowledge.

High-cardinality identifiers

Identifier question

Why might customer ID appear to create excellent training splits?

An identifier can isolate very small groups and memorize labels without learning reusable structure. Similar risks occur with post-outcome codes, timestamps that reveal the target period, and category levels seen only once.

Remove pure identifiers unless they have a defensible deployment role. Audit suspiciously strong features before celebrating model performance.

Computational behavior

At one node containing \(n\) rows and \(p\) features, sorting and scanning candidate numerical thresholds is roughly:

\[ O(p\,n\log n) \]

Intuition: each feature is ordered so adjacent values can become candidate boundaries, then the algorithm scans those boundaries. Across the complete tree, total fitting cost also depends on how many nodes grow and whether the tree is balanced. Worst-case analyses can be much higher; optimized implementations reduce practical cost by reusing sorted order and other cached work.

Prediction is simpler because one row follows only one root-to-leaf path:

\[ O(depth)\text{ per sample} \]

Intuition: trees can predict quickly because a row evaluates only the questions on its path, not every node.

Operational checks

Confirm all split features are available and trustworthy at prediction time.
Version the encoder, feature definitions, tree, and threshold together.
Monitor missingness, category frequencies, leaf traffic, probability calibration, and subgroup errors.
Alert when a formerly common leaf receives almost no rows or a rare leaf suddenly dominates.
Revalidate explanations after retraining because the tree structure can change sharply.