Skip to content

FRM Part I · FRM Exam Part I · Machine-Learning Methods

A bank builds a credit-scoring model. Preprocessing steps include standardizing features using the mean and standard deviation of the full dataset before splitting it into training and test sets, and the test set is then used repeatedly to choose among 50 candidate models. Which conclusion is most accurate?

The reported test error is likely optimistic because of data leakage from standardizing with full-sample statistics and from repeatedly using the test set to select among 50 models. Selection should use a separate validation set or cross-validation, keeping the test set untouched for the final assessment.

  1. AThe test-set error will be an unbiased estimate of out-of-sample performance because standardization is a linear transformation
  2. BThe procedure suffers from data leakage and selection on the test set, so the reported test error is likely optimisticCorrect
  3. CThe procedure is sound provided the test set contains at least 20% of the data
  4. DThe only problem is that standardization increases model variance, which makes test error pessimistic

Explanation

Standardizing with full-sample statistics lets test-set information influence training, which is leakage. Choosing among 50 models using the test set also means the test data guide selection, so the chosen model's test error is biased downward. A separate validation set or cross-validation should guide selection, leaving the test set untouched. Test-set size does not fix this.

Did you get it right without looking?

One question tells you little. A timed set on Machine-Learning Methods shows your real accuracy, how long you take and where you lose marks.

Practise Machine-Learning MethodsFree diagnostic mock

More Machine-Learning Methods questions