Skip to content

FRM Part II · FRM Exam Part II · Case Study: Model Risk and Model Validation

A bank's validation team is reviewing a new retail credit scoring model. The developers report excellent fit on the data used to build the model. Which validation technique most directly tests whether the model will generalise to new borrowers?

Out-of-sample testing is the most direct technique, because it assesses the model on data not used to build it. This reveals overfitting and shows whether performance holds for new borrowers, which in-sample fit measures cannot demonstrate.

  1. AOut-of-sample testing on data not used in model developmentCorrect
  2. BRecalculating the in-sample R-squared using the same data
  3. CReviewing the model documentation for formatting consistency
  4. DIncreasing the number of explanatory variables

Explanation

Out-of-sample testing evaluates performance on data excluded from estimation, which reveals overfitting and generalisation ability. In-sample fit statistics are expected to look good and do not test generalisation. Adding variables typically worsens overfitting.

Did you get it right without looking?

One question tells you little. A timed set on Case Study: Model Risk and Model Validation shows your real accuracy, how long you take and where you lose marks.

Practise Case Study: Model Risk and Model ValidationFree diagnostic mock

More Case Study: Model Risk and Model Validation questions