Concept 3

OLS and gradient descent

Both stories answer the same question: which slope and intercept produce the smallest overall error?

Ordinary Least Squares

OLS means choose the coefficients that minimize the sum of squared residuals.

\[\min_{\beta_0,\beta_1}\sum_{i=1}^{n}(y_i - \hat{y}_i)^2\]

For simple linear regression, the closed-form slope can be written as:

\[\beta_1 = \frac{\sum_{i=1}^{n}(x_i - \bar{x})(y_i - \bar{y})}{\sum_{i=1}^{n}(x_i - \bar{x})^2}\]\[\beta_0 = \bar{y} - \beta_1\bar{x}\]

Teacher framing: slope is based on how x and y move together, divided by how much x itself varies.

Concept check

If x and y usually increase together, should the slope be positive or negative?

Follow-up: What if x increases and y usually decreases?

Cost function intuition: move downhill until you reach the lowest error.

Gradient descent

Gradient descent does not jump directly to the answer. It starts with guesses for b0 and b1, checks the error, and repeatedly updates the guesses.

\[\theta_{new} = \theta_{old} - \alpha\nabla J(\theta)\]

Guess b0, b1 Predict y_hat Calculate MSE Find gradient Update values

Term	Meaning	Teaching warning
Gradient	Direction and steepness of change in cost.	It tells us how to move the parameter to reduce error.
Learning rate	Step size used during each update.	Too small is slow. Too large can overshoot.
Epoch	One full pass over the training data.	More epochs may help, but after convergence they add little value.
Convergence	Cost stops improving meaningfully.	The model has reached a stable minimum.

Human gradient descent

Tell students: imagine you are blindfolded on a hill and can only feel slope under your feet. How do you reach the lowest point?

Use their answers to explain gradient direction, step size, overshooting, and convergence.

Important distinction

Scikit-learn's standard LinearRegression uses a direct least-squares solver, not ordinary gradient descent. Still teach gradient descent because it becomes essential for neural networks and many large-scale optimization problems.