Skip to content

IAI Actuarial Core Principles · Risk Modelling and Survival Analysis · Elementary principles of machine learning

A data scientist at an Indian insurer tunes the regularisation parameter of a lasso regression. She tries many values and picks the one with the lowest error on the test set, then reports that test error as the expected future error. What is the main flaw?

The test set was used to select the regularisation parameter, so it is no longer an independent check. Its error is optimistically biased as an estimate of future performance. Tuning should use a validation set or cross-validation, leaving the test set for final assessment.

  1. AThe test set has influenced model selection, so its error is optimistically biased as an estimate of future performanceCorrect
  2. BLasso regression cannot be tuned using any held-out data
  3. CThe test error should be replaced by the training error, which is unbiased
  4. DUsing many parameter values always causes underfitting
  5. The test set should have been used for fitting the regression coefficients

Explanation

Once the test set is used to choose the parameter, it is no longer independent of the final model. Its error then understates true future error. A separate validation set or cross-validation should be used for tuning, keeping the test set for a single final assessment.

Did you get it right without looking?

One question tells you little. A timed set on Elementary principles of machine learning shows your real accuracy, how long you take and where you lose marks.

Practise Elementary principles of machine learningFree diagnostic mock

More Elementary principles of machine learning questions