Experiment 3 — Regularized Linear Regression

Lab Manual

Aim

To implement regularized linear regression (Ridge / L2 and Lasso / L1) in Python, understand how the regularization parameter controls overfitting, and compare the regularized models against ordinary least squares.

Objectives

Recall ordinary least-squares linear regression and the meaning of overfitting.
Add an L2 (Ridge) and L1 (Lasso) penalty to the cost function.
Study the effect of the regularization strength α (lambda) on the fit and on the coefficients.
Choose α using cross-validation and evaluate with train/test error.

Software

Python 3.x · Jupyter / Colab · NumPy, Pandas, Matplotlib, scikit-learn.

Theory

Linear regression minimises the sum of squared residuals. A flexible model (e.g. a high-degree polynomial) can fit the noise — overfitting. Regularization adds a penalty on the size of the coefficients, discouraging extreme weights. Ridge penalises Σwⱼ² (L2); Lasso penalises Σ|wⱼ| (L1), which can drive some coefficients to exactly zero.

Outcome

CO3 — building and comparing regression models on a dataset.

The Cost Function, in Words

Plain linear regression

Find weights w that make predictions close to reality — minimise the squared error:

cost = Σ (yᵢ − ŷᵢ)²

With enough flexibility, this chases every wiggle in the training data — including the noise.

+ Regularization penalty

Add a "keep the weights small" term. λ (alpha) sets how strict you are:

Ridge: Σ(yᵢ−ŷᵢ)² + λ·Σ wⱼ²
Lasso: Σ(yᵢ−ŷᵢ)² + λ·Σ |wⱼ|

λ = 0 → ordinary regression. λ → ∞ → weights crushed to ~0 (a flat line).

The Regularization Dial

A degree-9 polynomial is fit to noisy points with a real Ridge solver running live. Slide λ from left (no penalty) to right (strong penalty) and watch the purple curve go from over-fitted & wiggly → smooth & general → too flat. Green dots are training data, red rings are unseen test points.

Ridge fit · degree 9drag the slider ↓

Regularization λ λ = 0.0002

Train error

—

Test error

—

Weight size ‖w‖

—

Diagnosis

—

Watch the gap: when the train error is tiny but the test error is large, the model has memorised noise — that's overfitting. The right λ makes the test error smallest.

Ridge vs Lasso: the Big Difference

Both shrink coefficients as λ grows — but Lasso can hit exactly zero, dropping a feature entirely (automatic feature selection). Ridge only gets close to zero. Slide λ and watch the bars. (Shown for the clean orthonormal case, where the formulas are exact.)

Coefficient shrinkagegrey bar = coefficient removed (=0)

Regularization λ λ = 0.20

Notice: as you push λ right, Lasso's smaller coefficients snap to zero one by one, while Ridge's bars merely get shorter — never disappearing.

Watch the Concepts

Three short, beginner-friendly explainers from StatQuest with Josh Starmer — widely used in classrooms. Watch in order for a complete picture.

StatQuest · Part 1Ridge (L2) RegressionHow the L2 penalty desensitises the model to training data.

StatQuest · Part 2Lasso (L1) RegressionThe L1 penalty and how it removes useless variables.

StatQuest · VisualizedRidge vs Lasso, VisualizedWhy Lasso can set parameters to 0 and Ridge cannot.

Videos are embedded from YouTube and belong to their creators. If a frame is blank, your network may block YouTube — open the titles on youtube.com directly.

The Program

Implemented with scikit-learn. Swap in any dataset CSV (here a built-in one is used so it runs anywhere).

1 — data & split.py

import numpy as np
import pandas as pd
from sklearn.datasets import load_diabetes
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler

X, y = load_diabetes(return_X_y=True)   # or pd.read_csv("data.csv")
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, random_state=1)

# Always scale features before regularizing!
sc = StandardScaler().fit(X_tr)
X_tr, X_te = sc.transform(X_tr), sc.transform(X_te)

2 — ridge_and_lasso.py

from sklearn.linear_model import LinearRegression, Ridge, Lasso
from sklearn.metrics import mean_squared_error, r2_score

models = {
  "OLS":   LinearRegression(),
  "Ridge": Ridge(alpha=1.0),     # L2 penalty
  "Lasso": Lasso(alpha=0.1),     # L1 penalty
}
for name, m in models.items():
    m.fit(X_tr, y_tr)
    pred = m.predict(X_te)
    print(name, "MSE:", round(mean_squared_error(y_te, pred),1),
              "R2:", round(r2_score(y_te, pred),3))
    print("  non-zero coefs:", np.sum(m.coef_ != 0))

3 — choose_alpha_cv.py

# Pick the best alpha automatically with cross-validation
from sklearn.linear_model import RidgeCV, LassoCV

ridge_cv = RidgeCV(alphas=np.logspace(-3, 3, 50)).fit(X_tr, y_tr)
lasso_cv = LassoCV(cv=5).fit(X_tr, y_tr)

print("best Ridge alpha:", ridge_cv.alpha_)
print("best Lasso alpha:", lasso_cv.alpha_)

4 — plot_coefficients.py

import matplotlib.pyplot as plt

# See how coefficients shrink as alpha grows
alphas = np.logspace(-2, 3, 30)
coefs = [Ridge(alpha=a).fit(X_tr, y_tr).coef_ for a in alphas]

plt.plot(alphas, coefs)
plt.xscale("log"); plt.xlabel("alpha (λ)"); plt.ylabel("coefficient")
plt.title("Ridge coefficient shrinkage"); plt.show()

Procedure & Result

Steps followed

1 · Load & split
Read the data, split into train/test.

2 · Scale
Standardise features (mean 0, std 1).

3 · Baseline
Fit ordinary least squares (OLS).

4 · Regularize
Fit Ridge and Lasso with a chosen α.

5 · Tune
Use RidgeCV / LassoCV to pick α.

6 · Compare
Report MSE, R², and non-zero coefficients.

Result

Regularized linear regression was implemented using scikit-learn. Ridge and Lasso reduced overfitting compared with plain OLS, giving lower test error at a suitable α. Lasso additionally set several coefficients to exactly zero, performing automatic feature selection. This demonstrates CO3.

Conclusion: Regularization trades a little extra bias for a large drop in variance, producing models that generalise better to unseen data. λ is the control knob: too small overfits, too large underfits, and cross-validation finds the balance. Choose Ridge to keep all (correlated) features, Lasso when you also want a sparse, interpretable model.

Viva Questions

What is regularization and why do we need it?

Regularization adds a penalty on the size of the model's coefficients to the cost function. It discourages overly complex models that fit noise, reducing overfitting and improving generalisation to new data.

Difference between Ridge (L2) and Lasso (L1)?

Ridge adds λ·Σwⱼ² (squared coefficients) and shrinks all weights toward — but never exactly to — zero. Lasso adds λ·Σ|wⱼ| (absolute values) and can set some weights exactly to zero, performing feature selection.

What does the parameter α / λ control? What happens at the extremes?

It controls the strength of the penalty. λ = 0 reduces to ordinary least squares (may overfit); as λ → ∞ the coefficients are driven toward zero, giving an under-fitted, nearly flat model.

Why does Lasso produce sparse models but Ridge does not?

The L1 penalty has a corner (diamond-shaped constraint) at the axes, so the optimal solution often lands exactly on an axis where a coefficient is zero. The L2 penalty is circular/smooth, so solutions rarely hit zero exactly.

Why must features be standardised before regularization?

The penalty depends on coefficient magnitude, which depends on each feature's scale. Without standardisation, features measured in large units would be penalised differently, biasing the result.

What is the bias–variance trade-off?

Simple models have high bias but low variance; complex models have low bias but high variance. Total error is minimised at an intermediate complexity — exactly what tuning λ helps you find.

How do you choose the best α?

By cross-validation — e.g. RidgeCV / LassoCV try a range of α values and select the one with the lowest validation error.

What is Elastic Net?

A model that combines both penalties: λ₁·Σ|wⱼ| + λ₂·Σwⱼ². It gets Lasso's feature selection plus Ridge's stability with correlated features.

Turning the Knob
on Overfitting

The regularization idea