Classification from scratch

Logistic regression: from linear score to probability to decision.

Logistic regression is the bridge from regression thinking to classification. It starts with a linear score, converts it into a probability, learns using log loss, and makes decisions using a threshold.

Teaching story

Classification problem Linear score Sigmoid probability Log loss training Threshold and metrics

Opening question

If linear regression predicts 1.3 for “spam” and -0.2 for “not spam,” what should those numbers mean?

This motivates the need for probabilities between 0 and 1.

Session roadmap

Classification setup

Binary targets, examples, why linear regression is not enough, and the goal of predicting \(P(y=1\mid x)\).

Sigmoid and boundary

Linear score, sigmoid function, probability threshold, and decision boundary.

Loss and training

Why not MSE, binary cross entropy, cost function, gradient, and gradient descent.

Interpretation

Log-odds, odds ratios, coefficient meaning, regularization, and multiclass extension.

Evaluation

Confusion matrix, accuracy, precision, recall, F1, ROC-AUC, PR-AUC, threshold tuning, and imbalance.

Handling imbalance

Threshold tuning, class weights, oversampling, undersampling, SMOTE, stratification, and business-aware model selection.

Core formulas in one place

\[ z = \beta_0+\beta_1x_1+\cdots+\beta_px_p \qquad \hat{p}=\sigma(z)=\frac{1}{1+e^{-z}} \]

\[ L(y,\hat{p})=-\left[y\log(\hat{p})+(1-y)\log(1-\hat{p})\right] \]

\[ \nabla_\beta J=\frac{1}{n}X^T(\hat{p}-y) \qquad \beta := \beta-\alpha\nabla_\beta J \]

The session should feel familiar after linear regression: linear combination, cost function, gradient descent. The new pieces are sigmoid probability, log loss, and classification metrics.