FRM Exam Part I · Machine Learning and Prediction

Regularization in Regression: Ridge, LASSO and Elastic Net

Updated 11 October 2026 · Fact-checked

Regularization adds a penalty on coefficient size to the regression loss function. Ridge penalizes squared coefficients and shrinks them toward zero. LASSO penalizes absolute values and can set coefficients exactly to zero, selecting variables. Elastic net mixes both. You pick the penalty strength λ by cross-validation.

Understand Regularization: Ridge, LASSO and Elastic Net

Ordinary least squares picks coefficients that minimize the sum of squared residuals. With many correlated predictors, OLS fits noise. Coefficients become large and unstable, and the model does well in-sample but badly out-of-sample. This is overfitting.

Regularization fights overfitting by adding a penalty to the loss. The model now pays a price for large coefficients. It accepts a slightly worse in-sample fit (more bias) in return for more stable estimates (lower variance). This is the bias-variance tradeoff at work.

Ridge (L2) penalizes the sum of squared coefficients. It shrinks all coefficients toward zero but almost never sets any exactly to zero. It works well when many predictors each matter a little, and it handles multicollinearity by spreading weight across correlated variables.

LASSO (L1) penalizes the sum of absolute coefficients. The shape of this penalty lets some coefficients land exactly at zero, so LASSO does variable selection and gives a sparser, easier-to-explain model. With a group of highly correlated predictors, it tends to keep one and drop the others, and which one it keeps can be unstable.

Elastic net combines the L1 and L2 penalties. It selects variables like LASSO but handles correlated groups better, like ridge.

The tuning parameter λ sets the penalty strength. λ = 0 gives plain OLS. A larger λ means more shrinkage, more bias and less variance. You choose λ by cross-validation, not by in-sample fit. Predictors should be standardized first, otherwise the penalty treats variables with different units unfairly. The intercept is normally not penalized.

Key formulas to remember

OLS loss: Σ(yᵢ − ŷᵢ)²; Sum of squared residuals. Regularized models add a penalty to this.
Ridge objective: Σ(yᵢ − ŷᵢ)² + λ Σβⱼ²; L2 penalty. Shrinks coefficients but rarely makes them exactly zero. Intercept not penalized.
LASSO objective: Σ(yᵢ − ŷᵢ)² + λ Σ|βⱼ|; L1 penalty. Can set coefficients exactly to zero, so it selects variables.
Elastic net objective: Σ(yᵢ − ŷᵢ)² + λ₁ Σ|βⱼ| + λ₂ Σβⱼ²; Mix of L1 and L2. Some texts use one λ and a mixing weight instead.
Effect of λ: λ = 0 → OLS; λ ↑ → bias ↑, variance ↓; Choose λ by cross-validation on held-out data.

How to solve Regularization: Ridge, LASSO and Elastic Net questions

Use this method for any question on penalized regression, whether it is conceptual or numerical.

1Identify the penalty: squares of coefficients means ridge (L2); absolute values means LASSO (L1); both means elastic net.
2Check whether the question asks for variable selection or sparsity. If yes, LASSO or elastic net is the answer; ridge alone does not select.
3Check the data situation: many correlated predictors favours ridge or elastic net; a need for a simple, interpretable model favours LASSO.
4For numerical questions, compute the penalty term using the stated formula and add it to the sum of squared residuals.
5Track the direction of λ: larger λ gives more shrinkage, higher bias, lower variance and a simpler model.
6For tuning, choose λ by cross-validation error on validation data, never by training error.
7Confirm the answer: the intercept is not penalized, and predictors are standardized before fitting.

Quickest way: Penalty-shape shortcut

When to use it: Use it for conceptual multiple-choice questions where you must tell the three methods apart quickly.

Squared penalty: ridge. Shrinks, keeps all variables.
Absolute-value penalty: LASSO. Shrinks and drops some variables.
Both penalties: elastic net.
Any mention of choosing λ: the answer is cross-validation.
Any mention of larger λ: more bias, less variance.

Common mistakes in Regularization: Ridge, LASSO and Elastic Net

Saying ridge performs variable selection.
Ridge shrinks coefficients, and small values look like removal.
Fix: Ridge makes coefficients small but almost never exactly zero. Only LASSO and elastic net give exact zeros.
Choosing λ to minimize training error.
Training error always falls as λ falls, so λ = 0 looks best.
Fix: Pick λ using cross-validation or a validation set, which measures out-of-sample error.
Thinking a larger λ means a better-fitting model.
Students confuse a stronger penalty with stronger fitting.
Fix: Larger λ means more shrinkage: higher bias, lower variance, worse in-sample fit.
Forgetting to standardize predictors.
OLS is unaffected by units, so the issue seems new.
Fix: The penalty depends on coefficient size, which depends on units. Standardize predictors before fitting.
Including the intercept in the penalty.
The penalty sum is read as covering all parameters.
Fix: The sum runs over slope coefficients only. The intercept is normally left unpenalized.
Assuming LASSO is best with correlated predictors.
Sparsity sounds always desirable.
Fix: LASSO tends to pick one from a correlated group arbitrarily. Ridge or elastic net is more stable there.

Worked examples

Example 1

A ridge regression has fitted slope coefficients β₁ = 2, β₂ = −3 and β₃ = 1. The sum of squared residuals is 40 and λ = 5. Calculate the ridge objective value.

Show the solution

Ridge objective = Σ(yᵢ − ŷᵢ)² + λ Σβⱼ².
Sum of squared coefficients = 2² + (−3)² + 1² = 4 + 9 + 1 = 14.
Penalty = 5 × 14 = 70.
Objective = 40 + 70 = 110.

Answer: 110

Example 2

An analyst has 200 correlated candidate predictors of credit spreads and wants a simple model that uses only a subset. She fits LASSO and increases λ. Which statement is correct?

Show the solution

The goal is a subset of predictors, so variable selection is required.
LASSO uses an absolute-value penalty, which can set coefficients exactly to zero.
As λ increases, the penalty grows, more coefficients become exactly zero and the model gets simpler.
Bias rises and variance falls. The best λ is chosen by cross-validation, not by training error.

Answer: Increasing λ sets more coefficients to zero, giving a sparser model with higher bias and lower variance; choose λ by cross-validation.

Exam tips

Questions often test one contrast: ridge shrinks, LASSO shrinks and selects. Memorize it cold.
If a question says "sparse" or "fewer variables", the answer is LASSO or elastic net.
Always link λ to the bias-variance tradeoff: up means more bias, less variance.
Expect a simple penalty calculation: sum the squares or absolute values, multiply by λ, and add to the residual sum. Do not penalize the intercept unless told to.
Cross-validation is the default answer to "how should λ be chosen".

Practice questions from Machine Learning and Prediction

Regularization: Ridge, LASSO and Elastic Net: frequently asked questions

What is the main difference between LASSO and ridge regression?

Ridge penalizes the sum of squared coefficients and shrinks them without usually reaching zero. LASSO penalizes the sum of absolute coefficients and can set some exactly to zero. That makes LASSO a variable-selection method.

How does LASSO select variables?

The L1 penalty pushes weak predictors' coefficients all the way to zero as λ grows. Variables left with nonzero coefficients are the ones the model keeps.

How do I choose λ in ridge regression?

Use cross-validation. Fit the model across a range of λ values, measure error on held-out data, and pick the λ with the lowest out-of-sample error. Training error would always favour λ = 0.

When is elastic net better than LASSO?

Elastic net is useful when predictors are highly correlated. LASSO tends to keep one variable from a group and drop the rest, while the added ridge penalty in elastic net spreads weight more stably across the group.