Skip to content

FRM Part II · FRM Exam Part II · Credit Scoring and Rating

A bank's validation team finds that a probability of default model has strong out-of-sample AUC, but defaults in the most recent year were well above predictions in every grade, coinciding with a severe recession. The model was calibrated on a benign period using point-in-time PDs. Which conclusion is most appropriate?

Ranking ability appears intact given the high AUC, but calibration is deficient because defaults exceeded predictions across all grades in a downturn after calibration in benign conditions. The appropriate response is recalibration or adding a conservative margin, not rebuilding the ranking or ignoring calibration.

  1. ADiscriminatory power has failed, so the model must be rebuilt using different ranking variables
  2. BThe back-test is invalid because defaults are correlated, so no conclusion can be drawn
  3. CThe model should be judged acceptable since AUC is high, and calibration should be ignored
  4. DThe ranking ability seems intact but calibration is deficient, likely reflecting default correlation and the cycle, so recalibration or a conservative margin is warrantedCorrect

Explanation

High AUC shows the ranking is intact, while uniformly higher defaults across all grades point to a calibration shortfall tied to the cycle, as the long-run average calibrated in benign conditions understates stress-period PDs. Correlation makes binomial tests too aggressive, but it does not make the evidence meaningless. Ignoring calibration is wrong since PDs drive capital and pricing.

Did you get it right without looking?

One question tells you little. A timed set on Credit Scoring and Rating shows your real accuracy, how long you take and where you lose marks.

Practise Credit Scoring and RatingFree diagnostic mock

More Credit Scoring and Rating questions