Introduction to polynomial regression
Polynomial regression is the first major step beyond a straight-line model. It lets linear regression fit curved relationships by giving it transformed input features.
Why do we need it?
Simple linear regression assumes the prediction changes at a constant rate as \(x\) changes:
But many real relationships are not constant-rate relationships. The effect may increase, flatten, reverse, or saturate.
Real-world check
If study hours increase from 1 to 2, and then from 10 to 11, should the score improve by the same amount?
This question reveals why a straight line can be too rigid for many problems.
When the data bends, a straight line may miss the main pattern even if it is mathematically correct.
What is polynomial regression?
Polynomial regression extends linear regression by adding powers of the input feature.
The degree \(d\) controls how flexible the curve can be.
Under squared-error loss, the ideal regression prediction is the conditional average of \(y\) at each value of \(x\):
That distinction matters because noisy points may sit above or below the underlying average pattern. The fitted curve should usually follow the pattern, not every noisy point.
| Degree | Model | Shape it can learn |
|---|---|---|
| 1 | \(\beta_0+\beta_1x\) | Straight line |
| 2 | \(\beta_0+\beta_1x+\beta_2x^2\) | One bend, U-shape or inverted U-shape |
| 3 | \(\beta_0+\beta_1x+\beta_2x^2+\beta_3x^3\) | More flexible curve |
| High degree | Many powers of \(x\) | Very flexible, but can overfit noise |
Curve fitting, interpolation, and smoothing
Polynomial regression belongs to a larger family of curve fitting ideas. The important classroom distinction is whether we want an exact fit or an approximate fit.
| Idea | Meaning | Polynomial regression connection |
|---|---|---|
| Interpolation | The curve is forced to pass through known points. | Can look perfect on training points but may become unstable between points. |
| Smoothing | The curve follows the main trend while ignoring some noise. | Usually closer to the ML goal when data is noisy. |
| Least-squares fitting | The model chooses the curve with small total squared vertical errors. | This is the usual way polynomial regression learns coefficients. |
Fitting choice
If the data contains measurement noise, should our curve interpolate every point or smooth the overall trend?
For prediction, smoothing the trend is usually safer than forcing an exact fit.
How can it be linear and curved?
The model is linear in the coefficients, not necessarily linear in the original input.
The term \(x^2\) creates curvature, but the model still learns coefficients using linear regression methods.
The effect of \(x\) is no longer constant. For a quadratic model, the slope at a particular \(x\) is:
So a one-unit change in \(x\) can have a different impact at low, medium, and high values of \(x\).
This is the key trick: transform the inputs, then fit a linear model in the transformed feature space.
Polynomial regression is feature engineering plus linear regression.
Examples where polynomial features help
Experience vs salary
Salary may rise quickly early in a career and later flatten.
Ad spend vs sales
More spend can help at first, but returns may diminish after a point.
Medicine dosage
Response may improve up to a dose, then worsen or saturate.
Feature decision
If adding \(x^2\) improves the fit, should we immediately add \(x^3,x^4,\ldots,x^{20}\)?
This sets up the next part of the story: flexibility can help, but too much flexibility can memorize noise.