Skip to content

IAI Actuarial Core Principles · Risk Modelling and Survival Analysis · Elementary principles of machine learning

When a validation set is used to choose between several candidate models and the chosen model is then assessed, why is a separate final test set still recommended?

A separate test set is needed because the selected model was chosen for its low validation error, making that figure optimistically biased. An untouched test set gives an unbiased estimate of how the model will perform on new data.

  1. AThe validation error of the selected model is optimistically biased because it was chosen for its low validation errorCorrect
  2. BThe validation set is always larger than the training set
  3. CA test set removes the need for any training data
  4. DValidation sets can only be used to estimate bias, not error
  5. The test set is used to fit the model parameters

Explanation

Picking the model with the lowest validation error means that error is selected partly by chance and so understates true error. An untouched test set gives an unbiased estimate of generalisation performance. The test set is not used for fitting or selection.

Did you get it right without looking?

One question tells you little. A timed set on Elementary principles of machine learning shows your real accuracy, how long you take and where you lose marks.

Practise Elementary principles of machine learningFree diagnostic mock

More Elementary principles of machine learning questions