Skip to content

FRM Part I · FRM Exam Part I · Machine Learning and Prediction

An analyst has 1,000 monthly observations and uses a 60/20/20 split into training, validation and test sets. Which statement about the use of these subsets is correct?

The validation set is used to tune hyperparameters and choose among models, while the test set is used only once at the end to estimate final out-of-sample performance. Reusing the test set for tuning leaks information and gives an overly optimistic performance estimate.

  1. AThe test set should be used repeatedly to tune hyperparameters, since it is unseen by the training step
  2. BThe validation set is used to tune hyperparameters and select the model, and the test set is used once to estimate final out-of-sample performanceCorrect
  3. CThe training set is used to estimate final out-of-sample performance, and the validation set to fit parameters
  4. DThe validation set and test set should be merged into the training set before model selection to reduce bias

Explanation

Parameters are fitted on the training set, hyperparameters and model choice are tuned on the validation set, and the test set is kept untouched until the end to give an unbiased estimate of out-of-sample performance. Tuning on the test set would leak information and make the estimate optimistic.

Did you get it right without looking?

One question tells you little. A timed set on Machine Learning and Prediction shows your real accuracy, how long you take and where you lose marks.

Practise Machine Learning and PredictionFree diagnostic mock

More Machine Learning and Prediction questions