FRM Part II · FRM Exam Part II

Case Study: Model Risk and Model Validation: formula sheet

Full chapter guide

Key formulas

Definition of model risk (SR 11-7 style): Model risk = risk of adverse consequences from decisions based on incorrect or misused model outputs; Consequences include financial loss, poor business and strategic decisions, and damage to reputation.
Two main sources: Model risk = fundamental errors + incorrect or inappropriate use; Fundamental errors cover assumptions, data, implementation and calibration. Use covers outside-scope application and poor oversight.
Risk grows with: Higher model risk ← greater complexity, greater uncertainty of inputs, broader use, larger potential impact; This is a general principle, not a numerical formula. Use it to justify more intensive validation.
Model risk sources: Model risk = model error (design, data, implementation) + model misuse (use outside purpose); Use this split to classify any case failure.
Effective challenge: Effective challenge = competent, independent, influential review with authority to require change; The core control in model governance (SR 11-7 language).
Loss vs VaR check: Exception if actual loss > VaR at the stated confidence level; At 99% one-day VaR, expect about 1% of days to be exceptions.
Leverage effect: Return on equity ≈ asset return × (Assets ÷ Equity) − funding cost effect; High leverage turns a small asset loss into a large equity loss, as at LTCM.
Three core elements of validation (SR 11-7): Validation = conceptual soundness + ongoing monitoring + outcomes analysis; Backtesting sits inside outcomes analysis. Benchmarking is a supporting comparison tool.
Backtest exception rate: Exception rate = number of exceptions ÷ number of observations; Compare with the expected rate, 1 − confidence level. For 99% VaR, expect 1%.
Expected exceptions: Expected exceptions = N × (1 − c); N is the number of days and c the VaR confidence level. A large gap from this value suggests model weakness.
Verification vs validation: Verification: built as intended? Validation: fit for purpose?; Verification is narrower and focuses on implementation and calculation accuracy.
Definition of model risk: Model risk = risk from (1) fundamental model errors + (2) incorrect or inappropriate use of the model; Two sources. Use is as important as the build.
Three pillars of SR 11-7: Development, implementation and use + Validation + Governance, policies and controls; Most questions fit one of these pillars.
Core validation elements: Conceptual soundness + Ongoing monitoring (including benchmarking) + Outcomes analysis (including backtesting); Validation is more than backtesting. Frequency is at least annual review in practice, set by bank policy.
Effective challenge: Competence + Influence + Incentives + Independence from development; Without influence, validators cannot force fixes.
Model inventory contents: Purpose, owner, users, inputs, limitations, validation status and dates, vendor or in-house; The inventory covers all models, including vendor models and those under development.
Model range (uncertainty spread): Range = Max(model values) − Min(model values); Simple measure of uncertainty across competing models. It ignores how likely each model is.
Model reserve from a range: Reserve = Reported value (or mid-model value) − Conservative value; Policies often set the reserve as some fraction of the range. Use the fraction the question states.
Model-risk adjusted risk measure: Adjusted VaR = Base VaR × (1 + add-on %); A multiplicative overlay. For example, a 10% add-on turns a 20 into 22.
Sensitivity of output to a parameter: Sensitivity = Change in output ÷ Change in parameter; A large value means the output depends heavily on a hard-to-estimate input.
Capital versus reserve: Reserve covers expected valuation error; capital buffer covers unexpected model error; A rule to recall for interpretation questions.

Quick revision

Model risk is loss or bad decisions from a model that is wrong, misused or misunderstood.
Main sources: poor data, wrong specification or assumptions, implementation errors, and misuse outside intended scope.
A model can be sound in theory and still fail because of how it is used.
London Whale: a VaR model change and weak oversight let the risk in the credit portfolio be understated.
LTCM: models relied on historical relationships and assumed enough liquidity, and leverage magnified the losses when markets moved against it.
Validation includes conceptual soundness review, data and implementation checks, and outcomes analysis such as backtesting.
Benchmarking compares model output with an alternative model or an independent estimate.
Validation must be independent of model development and be able to give effective challenge.
SR 11-7 expects a model inventory, clear roles, documentation and ongoing monitoring.
Senior management and the board are responsible for the overall model risk framework.
Mitigation includes limits, conservative adjustments, overlays and use restrictions.
Quantifying model risk is hard; common approaches use alternative models, parameter uncertainty and stress or scenario analysis.

Common mistakes

Calling every model failure an assumption error. Fix: Check whether the theory is sound but the data or code is wrong. Pick the bucket that matches the stated cause.
Treating a good model used wrongly as a fundamental error. Fix: If the model works for its intended purpose but was applied to a new product or market, the source is misuse.
Mixing up the two cases, for example saying LTCM changed its VaR model to fit limits. Fix: Tie one tag to each: London Whale means VaR model change and weak validation; LTCM means leverage, correlation and liquidity assumptions.
Saying the failure shows VaR is useless. Fix: The lesson is that VaR must be validated, backtested and supplemented with stress tests, not dropped.
Treating verification and validation as the same thing Fix: Verification checks implementation against intent. Validation also tests conceptual soundness and fitness for use.
Saying backtesting is a separate element from outcomes analysis Fix: Treat backtesting as one type of outcomes analysis under SR 11-7.
Treating validation as only backtesting. Fix: Remember the three elements: conceptual soundness, ongoing monitoring and outcomes analysis.
Letting the model developer validate their own model. Fix: Validation needs independence and effective challenge. Developers can test, but that is not validation.
Saying model risk can be eliminated by validation. Fix: Validation reduces and reveals model risk. Residual risk always remains and needs limits, adjustments or buffers.
Confusing a model reserve with regulatory capital. Fix: A reserve reduces valuation or earnings for expected uncertainty. Capital absorbs unexpected losses.

Exam tips

Scenario questions usually hide one clue word: stale, spreadsheet, outside scope, assumed normal. Match it to the source.
Distinguish fundamental errors from misuse. Many wrong options swap the two.
Expect governance links: validation independence, inventory and documentation are common answers.
Do not say model risk is eliminated. Correct options say reduced or managed.
Know that SR 11-7 is the reference framework GARP uses for model risk definitions.
Know one-line summaries of each case and the specific model flaw: VaR model change and validation weakness for the Whale; leverage, correlation and liquidity for LTCM.
Questions often ask for the control that was missing. Answer with independent validation, effective challenge, stress testing or escalation.
Do not memorise exact loss amounts. Focus on causes and lessons, which are what MCQs test.