Regularization & the Bias–Variance Tradeoff

Two sources of error: bias and variance

Earlier you met overfitting and underfitting. The bias–variance tradeoff is the formal framework behind them — the language professionals use to diagnose a model.

• Bias = error from being too simple. A high-bias model makes the same kind of mistake everywhere because it cannot capture the real pattern. High bias = underfitting.
• Variance = error from being too sensitive. A high-variance model changes wildly with small data changes because it chases noise. High variance = overfitting.

A dartboard analogy: high bias is darts tightly grouped but far from the bullseye (consistently wrong). High variance is darts scattered everywhere (all over the place). You want them grouped near the centre — low bias and low variance. The tradeoff is that pushing one down often nudges the other up, so you seek the sweet spot.

Regularization: a penalty for complexity

Regularization is the standard cure for high variance (overfitting). The idea is simple: add a penalty to the cost function for making the model too complex — specifically for letting its coefficients (the per-feature weights) grow large. The model must now balance fitting the data against keeping its weights small, so it stops chasing noise.

• L2 / Ridge — penalises the sum of squared weights. It shrinks all weights toward zero but rarely to exactly zero. Great general-purpose default.
• L1 / Lasso — penalises the sum of absolute weights. It can drive some weights exactly to zero, effectively deleting useless features — so it doubles as automatic feature selection.

A worked example: Ridge and Lasso shrink the weights

We make data where only the first feature truly matters and the other two are noise, then compare plain regression with Ridge and Lasso. The strength of the penalty is set by alpha — bigger alpha means a harsher penalty.

from sklearn.linear_model import LinearRegression, Ridge, Lasso

# feature 1 drives y; features 2 and 3 are random noise
X = [[1, 5, 9], [2, 1, 3], [3, 8, 2], [4, 2, 7], [5, 6, 1]]
y = [10, 20, 30, 40, 50]      # y is basically 10 * feature1

for name, model in [('Plain', LinearRegression()),
                    ('Ridge', Ridge(alpha=1.0)),
                    ('Lasso', Lasso(alpha=1.0))]:
    model.fit(X, y)
    print(name, 'weights:', [round(c, 2) for c in model.coef_])

Why does this help? By refusing to let the noise features grab large weights, the regularized models avoid memorising flukes in this tiny dataset — so they generalise better to new rows. The bigger you make alpha, the stronger the pull toward small (or zero) weights.