Skip to content

FRM Part II · FRM Exam Part II · Supervisory Guidance on Model Risk Management

A quant team develops a default-prediction model and tunes it on a dataset of 2010-2019 loans. It reports a very high accuracy on that same dataset and proposes to go live. Which development weakness is MOST evident?

The most evident weakness is the absence of out-of-sample testing, which creates overfitting risk. Accuracy measured on the same data used for tuning overstates predictive power, so guidance expects holdout or out-of-time testing to show the model generalizes before it goes live.

  1. ALack of out-of-sample testing, creating a risk of overfittingCorrect
  2. BExcessive use of conservative assumptions
  3. COverreliance on expert judgment in the data selection
  4. DUse of too many validation staff

Explanation

Performance measured on the data used for calibration is biased upward and gives no evidence of predictive ability. Guidance expects out-of-sample and out-of-time testing to detect overfitting. Conservatism or staffing levels are not the issue shown.

Did you get it right without looking?

One question tells you little. A timed set on Supervisory Guidance on Model Risk Management shows your real accuracy, how long you take and where you lose marks.

Practise Supervisory Guidance on Model Risk ManagementFree diagnostic mock

More Supervisory Guidance on Model Risk Management questions