Part 3

Soft margin SVM accepts that real data is messy.

Hard margin is elegant, but real datasets overlap. Soft margin allows controlled violations so the model can generalize better.

Why hard margin fails

If one noisy point appears on the wrong side, should the whole boundary twist around it?

In real data, classes can overlap. A perfectly separating boundary may not exist, or may overfit badly.

Soft margin says: allow some points to violate the margin, but charge a penalty.

How can we measure how much a point violates the margin?

Soft margin introduces slack variables \(\xi_i\). A slack value measures how much point \(i\) violates the margin condition.

\[ y_i(w^Tx_i+b)\ge 1-\xi_i \qquad \xi_i\ge 0 \]

Slack value	Meaning
\(\xi_i=0\)	point is correctly classified and outside margin
\(0<\xi_i<1\)	point is inside margin but still on correct side
\(\xi_i\ge 1\)	point is misclassified or exactly beyond the boundary

Should the model prefer a wide margin or fewer violations?

\[ \min_{w,b}\frac{1}{2}\|w\|^2+C\sum_i\xi_i \]

The first term wants a wide margin. The second term penalizes violations. \(C\) controls how much violations hurt.

What happens if mistakes become very expensive?

Value of \(C\)	Behavior	Risk
Small \(C\)	more tolerant, wider margin, allows more violations	underfitting
Large \(C\)	stricter, tries to reduce training violations	overfitting

If one class is rarer but important, should mistakes on both classes cost the same?

SVM can use class weights so minority-class mistakes become more expensive.

SVC(class_weight="balanced")

In sklearn, balanced class weights are roughly inversely proportional to class frequency.

\[ \text{class weight}_c \propto \frac{1}{\text{frequency of class }c} \]

This does not magically solve imbalance. It changes the cost of errors, so evaluate precision, recall, and F1 carefully.