Skip to content

FRM Part II · FRM Exam Part II · Case Study: Model Risk and Model Validation

A bank's internal PD model shows strong ranking power in development data but the validator observes that the validation sample comes from the same period and portfolio used to fit the model. Which is the most appropriate concern and remedy?

The concern is overfitting: validating on the same data used to fit the model overstates performance. The remedy is out-of-sample and out-of-time testing on observations and periods not used in development. Recalibration, rebuilding data systems or switching to a vendor model do not fix the flawed validation design.

  1. AOverfitting risk; use out-of-sample and out-of-time testingCorrect
  2. BModel drift; recalibrate monthly with the latest data
  3. CData lineage failure; rebuild the data warehouse
  4. DBenchmark mismatch; adopt the vendor model

Explanation

Testing on the fitting data overstates performance because the model may have captured sample noise. Out-of-sample (different observations) and out-of-time (different period) tests give a fairer view of predictive power. Recalibrating or replacing the model does not address the flawed test design.

Did you get it right without looking?

One question tells you little. A timed set on Case Study: Model Risk and Model Validation shows your real accuracy, how long you take and where you lose marks.

Practise Case Study: Model Risk and Model ValidationFree diagnostic mock

More Case Study: Model Risk and Model Validation questions