FRM Exam Part I · Machine Learning and Prediction

Overfitting, Bias-Variance Tradeoff and Data Splitting

Updated 11 October 2026 · Fact-checked

Overfitting means a model fits noise in the training data and predicts new data poorly. Underfitting means it is too simple to capture the pattern. The bias-variance tradeoff balances the two. You tune complexity on a validation set or by cross-validation, then judge final performance once on a test set.

Understand Overfitting, Bias-Variance Tradeoff and Data Splitting

A machine learning model is built to predict data it has not seen. It is fitted on a training sample, but what matters is how it performs out of sample. A model can look excellent on the training data and still fail on new data.

Overfitting happens when a model is too flexible. It learns the random noise in the training data as if it were signal. Training error is very low, but error on new data is high. Examples are a regression with too many variables, a very deep decision tree, or a high-degree polynomial. Underfitting is the opposite. The model is too simple, misses real structure, and has high error on both training and new data.

The bias-variance tradeoff explains this. Bias is error from wrong or overly simple assumptions, so the model is systematically off. Variance is how much the fitted model changes if you train it on a different sample. Simple models have high bias and low variance. Complex models have low bias and high variance. Expected out-of-sample squared error is roughly bias squared plus variance plus irreducible noise. As complexity rises, training error keeps falling, but out-of-sample error falls and then rises. You want the complexity near the bottom of that U shape.

To measure this honestly you split the data. The training set fits the model parameters. The validation set compares models and tunes hyperparameters, such as the penalty strength, tree depth or number of neighbors. The test set is held back and used once at the end to estimate real performance. If you tune on the test set, it stops being a fair check.

When data is limited, use k-fold cross-validation. Split the data into k equal folds. Train on k − 1 folds and validate on the remaining one. Repeat k times so each fold is the validation set once, then average the k errors. With time series data, such as financial returns, you must not shuffle randomly. Keep the time order, and train on the past and validate on the future, to avoid look-ahead bias.

Key formulas to remember

Error decomposition: Expected out-of-sample squared error = Bias² + Variance + Irreducible error; Irreducible error is noise you cannot remove by any model. Bias and variance are the parts you can trade off.
Complexity effect: More complexity → Bias ↓, Variance ↑ (and the reverse); Training error always tends to fall with complexity. Validation error is U-shaped.
k-fold cross-validation error: CV error = (1 ÷ k) × Σ (error on fold i), i = 1 to k; Each observation is used for validation exactly once and for training k − 1 times.
Training size in k-fold: Training observations per round = N × (k − 1) ÷ k; validation = N ÷ k; Leave-one-out is the case k = N.

How to solve Overfitting, Bias-Variance Tradeoff and Data Splitting questions

Use this approach for any question on fit, complexity or data splitting.

1Identify what the question asks: diagnosing a fit problem, choosing a split, or computing a cross-validation result.
2Compare training error and validation error. Both high suggests underfitting. Low training error with much higher validation error suggests overfitting.
3Link the diagnosis to bias and variance. Underfitting means high bias and low variance. Overfitting means low bias and high variance.
4Pick the remedy. For overfitting: simpler model, more data, regularization, fewer features, or early stopping. For underfitting: more flexibility or better features.
5Check which sample does which job: training fits parameters, validation tunes hyperparameters and selects models, test gives the final unbiased estimate.
6For k-fold questions, compute fold sizes with N ÷ k and average the k errors, choosing the lowest average error as the best setting.
7For time series, confirm the split respects time order, with no future data in training.

Quickest way: Error-pattern shortcut

When to use it: Use it when the question gives training and validation errors or asks which problem or sample is involved.

Training error high and validation error similar: underfit, high bias.
Training error low and validation error much higher: overfit, high variance.
Tuning or selecting a model: use the validation set or cross-validation.
Final performance claim: test set, used once.
Pick the model with the lowest average validation error, not the lowest training error.

Common mistakes in Overfitting, Bias-Variance Tradeoff and Data Splitting

Choosing the model with the lowest training error.
Training error always improves with more complexity, so it looks like progress.
Fix: Select on validation or cross-validation error. Training error is not a measure of predictive power.
Using the test set to tune hyperparameters.
Students think any held-out data can be reused.
Fix: Tune on validation data. Touch the test set once at the end, otherwise its estimate is optimistic.
Mixing up bias and variance.
Both sound like generic error.
Fix: Bias is systematic error from a too-simple model. Variance is sensitivity to the particular training sample. Overfit means high variance.
Shuffling time series data randomly before splitting.
Standard k-fold is taught with random folds.
Fix: Keep chronological order so the model never trains on information from after the validation period.
Thinking more data always fixes underfitting.
More data helps variance, so it seems a universal cure.
Fix: Underfitting is a bias problem. A simple model stays too simple with more data. Add flexibility or better features.

Worked examples

Example 1

A model has a training error of 2% and a validation error of 11%. A simpler version has a training error of 7% and a validation error of 8%. Which model is preferred and why?

Show the solution

The first model has a big gap between training and validation error, which signals overfitting and high variance.
The second model has a small gap and the lower validation error, 8% against 11%.
Model choice uses validation error, not training error.
So prefer the simpler model.

Answer: The simpler model, because it has the lower validation error (8% versus 11%). The first model is overfit.

Example 2

You have 1,000 observations and run 5-fold cross-validation for a penalty setting. The validation errors on the folds are 4.0, 5.0, 6.0, 4.5 and 5.5. How many observations are in each validation fold and in each training set, and what is the cross-validation error?

Show the solution

Validation fold size = N ÷ k = 1,000 ÷ 5 = 200.
Training size = 1,000 × (5 − 1) ÷ 5 = 800.
Sum of errors = 4.0 + 5.0 + 6.0 + 4.5 + 5.5 = 25.0.
CV error = 25.0 ÷ 5 = 5.0.

Answer: Each validation fold has 200 observations, each training set has 800, and the cross-validation error is 5.0.

Exam tips

Expect conceptual questions that give a pair of training and validation errors and ask you to diagnose overfitting or underfitting.
Memorize the role of each sample: training fits, validation tunes, test confirms once.
Be ready for simple k-fold arithmetic: fold sizes and the average of fold errors.
Remember that regularization, simpler models and more data reduce variance, and a note on time series splits is a likely trap.
When an option says to pick the model with the best training fit, it is almost always wrong.

Practice questions from Machine Learning and Prediction

Overfitting, Bias-Variance Tradeoff and Data Splitting: frequently asked questions

What is the difference between overfitting and underfitting?

Overfitting means the model captures noise, so training error is low but out-of-sample error is high. Underfitting means the model is too simple, so error is high on both training and new data.

What is the bias-variance tradeoff in simple terms?

Simple models are consistently off (high bias) but stable (low variance). Complex models fit the training data closely (low bias) but change a lot between samples (high variance). You choose complexity to minimize total out-of-sample error.

How does k-fold cross-validation work?

You split the data into k equal folds. Each round trains on k − 1 folds and validates on the one left out. After k rounds you average the validation errors to compare models or hyperparameters.

What is the difference between the validation set and the test set?

The validation set is used repeatedly to tune hyperparameters and choose between models. The test set is kept aside and used once at the end to give an unbiased estimate of performance on new data.