Part 4 · Binary classification

A classification model still adds numerical corrections.

Gradient Boosting does not ask each tree to make a final yes-or-no decision. It builds a continuous raw score, converts that score to a probability, measures log loss, and uses a regression tree to improve the scores.

Prediction question

If trees output corrections such as \(+0.55\) and \(-0.55\), how can their sum become a class probability?

Three objects must stay separate

Raw score \(F(x)\)

The additive quantity built by the trees. It can be any real number.

Probability \(p(x)\)

The sigmoid transforms the raw score into a number between 0 and 1.

\[p(x)=\sigma(F(x))=\frac{1}{1+e^{-F(x)}}\]

Predicted class

A threshold turns the probability into a decision, commonly \(\widehat y=1\) when \(p\geq0.5\).

Training loss

Binary log loss judges the probability, not merely the final class label.

\[L(y,p)=-\left[y\log p+(1-y)\log(1-p)\right]\]

Why use a raw score? Sums are unrestricted, but probabilities must remain in \([0,1]\). Add trees on the raw-score scale and use the sigmoid only to interpret the result as probability.

Baseline question

Before using any feature, what probability should the model assign to every application?

Step 0: start from the class prior

Suppose eight historical loan applications are ordered by a risk feature. The target is \(y=1\) for default and \(y=0\) for repayment.

Risk rank \(x\)12345678
Actual \(y\)10110010

There are four defaults in eight rows, so the initial probability is

\[p_0=\frac{4}{8}=0.5\]

Why convert probability into a raw score?

First convert probability to odds. Odds compare the chance of default with the chance of repayment:

\[\operatorname{odds}=\frac{p_0}{1-p_0}=\frac{0.5}{0.5}=1\]

Odds of \(1\) mean \(1{:}1\): the two outcomes are equally likely. The raw score is the logarithm of these odds:

\[F_0=\log\left(\frac{p_0}{1-p_0}\right)=\log(1)=0\]

Raw scores can range from \(-\infty\) to \(+\infty\), so trees can safely add corrections to them. The sigmoid converts the score back to probability:

\[\sigma(F_0)=\frac{1}{1+e^{-0}}=0.5\]

Score direction: \(F=0\) means \(p=0.5\); a positive score means \(p>0.5\); and a negative score means \(p<0.5\).

Why is the initial mean log loss \(0.6931\)?

The model has not used any features yet, so every row receives probability \(0.5\). For a default row \((y=1)\):

\[L(1,0.5)=-\log(0.5)=0.6931\]

For a repayment row \((y=0)\):

\[L(0,0.5)=-\log(1-0.5)=0.6931\]

All eight rows therefore have the same loss, so averaging them gives the same number:

\[\operatorname{LogLoss}_0=\frac{8(0.6931)}{8}=0.6931\]

Starting point: before seeing any features, the model predicts only the overall class proportion. The trees must move individual raw scores above or below zero to improve this baseline.

General baseline: if the positive-class fraction is \(\pi\), then \(F_0=\log(\pi/(1-\pi))\). An imbalanced dataset therefore does not normally begin at probability \(0.5\).

Correction question

For a positive row currently predicted as \(0.5\), should its probability move up or down? What about a negative row?

Step 1: calculate the classification pseudo-residuals

For binary log loss, differentiating with respect to the raw score gives

\[\frac{\partial L}{\partial F}=p-y\qquad\Longrightarrow\qquad r_i=-\frac{\partial L}{\partial F}=y_i-p_i\]

Because every initial probability is \(0.5\), positives receive \(+0.5\) and negatives receive \(-0.5\).

\(x\)\(y\)Current \(p_0\)Pseudo-residual \(r=y-p_0\)Meaning
110.5+0.5Push upward
200.5−0.5Push downward
310.5+0.5Push upward
410.5+0.5Push upward
500.5−0.5Push downward
600.5−0.5Push downward
710.5+0.5Push upward
800.5−0.5Push downward

The tree's target is continuous: \(r_i\) says both direction and urgency. This is why Gradient Boosting uses a regression tree internally even for classification.

Split question

Looking only at the \(+0.5\) and \(-0.5\) targets, where would a one-split tree divide the rows?

Step 2: fit a small regression tree to the corrections

Among the seven possible split positions, the smallest squared error occurs between ranks 4 and 5. The first leaf contains three positives and one negative; the second contains one positive and three negatives.

split at 4.5leaf A: ranks 1–4leaf B: ranks 5–8+0.5−0.5
positive rownegative row

What the split discovered

Leaf A has observed default rate \(3/4=0.75\). Leaf B has observed default rate \(1/4=0.25\).

The stump has found a useful regional pattern, though it is not perfect. That is exactly the role of a weak learner.

\[\bar r_A=+0.25,\qquad \bar r_B=-0.25\]

These means guide the split search. After fixing the regions, Gradient Boosting chooses leaf updates that best reduce the actual log loss.

Update question

Should the next probability be obtained by adding \(0.25\) directly to \(0.5\), or by updating the raw score?

Step 3: find each leaf's best raw-score correction

Inside a leaf with positive fraction \(q\), the log-loss-minimizing raw score is its log-odds, \(\log(q/(1-q))\). Since the current score is zero:

\[\gamma_A=\log\left(\frac{0.75}{0.25}\right)=\log 3\approx1.0986\]
\[\gamma_B=\log\left(\frac{0.25}{0.75}\right)=-\log 3\approx-1.0986\]

Use learning rate \(\eta=0.5\), so the model takes only half of each correction:

\[F_1(x)=F_0(x)+0.5\,\gamma_{\operatorname{leaf}(x)}\]
RegionNew raw score \(F_1\)New probability \(\sigma(F_1)\)Direction
Ranks 1–4+0.54930.6340More confidence in default
Ranks 5–8−0.54930.3660More confidence in repayment

Do not add corrections directly to probabilities. Repeated probability additions could leave the valid range. Corrections are added to raw scores; the sigmoid always returns a valid probability.

This page uses exact leaf-wise log-loss minimization so the arithmetic stays transparent. Some implementations use a Newton approximation to obtain closely related leaf updates.

Verification question

The tree still misclassifies two rows at a \(0.5\) threshold. Can log loss nevertheless improve?

Step 4: verify that the loss fell

Six rows agree with their leaf's majority and receive probability \(0.6340\) for their true class. Two disagree and receive probability \(0.3660\) for their true class. Therefore:

\[\begin{aligned}\operatorname{LogLoss}_1&=-\frac18\left[6\log(0.6340)+2\log(0.3660)\right]\\&\approx0.5930\end{aligned}\]

Before the tree

Mean log loss = 0.6931

After one cautious tree

Mean log loss ≈ 0.5930

The hard-label accuracy rises from an ambiguous \(0.5\)-probability baseline to \(6/8=75\%\). More importantly, the differentiable log loss supplies graded feedback for the next round. The model recalculates \(y-p_1\), fits another correction tree, and continues.

The classification loop: raw scores → sigmoid probabilities → log loss → pseudo-residuals \(y-p\) → regression tree → raw-score correction → repeat.

What this example establishes

  1. Classification Gradient Boosting builds scores before it produces classes.
  2. The initial score represents the class prior.
  3. For binary log loss, the negative-gradient target is \(y-p\).
  4. A regression tree groups observations needing similar score corrections.
  5. Leaf values update raw scores, and the sigmoid converts them to probabilities.
  6. A weak tree need not classify every row correctly to reduce the ensemble's loss.
Previous: Why gradient?Next: Algorithm