Skip to content

CFA Level II · CFA Level II Exam

Backtesting and Simulation for CFA Level II

Backtesting applies an investment rule to past data to see how it would have performed. Simulation generates many possible future outcomes using historical data, assumed distributions, or random draws. To solve questions, identify the method, check the inputs and assumptions in the vignette, spot biases, and judge how reliable the results are.

What this chapter covers

This chapter is about testing ideas before you trust them. Backtesting asks how a rule or strategy would have performed on past data. Simulation asks what range of outcomes could occur in the future. Both produce numbers that look precise, so the exam rewards your ability to question where those numbers came from.

You will meet three simulation approaches: historical simulation, which resamples actual past returns; parametric simulation, which assumes a distribution and fitted parameters such as mean, standard deviation and correlation; and Monte Carlo simulation, which draws random values from a specified model, often over many periods and many assets. The chapter also covers common pitfalls such as look-ahead bias, survivorship bias, data snooping and overfitting, plus the limits of any model-based result.

The chapter connects to much of the paper. Quantitative Methods supplies the statistics and the idea of out-of-sample testing. Portfolio Construction uses simulation for retirement and goal planning, asset allocation and risk estimates. Derivatives and Risk Management uses the same logic for VaR and scenario analysis. Ethics also links in, since presenting a cherry-picked backtest as proof can mislead clients. In an item set, you usually get a vignette describing a process and must judge whether it is sound.

Questions on this chapter are mostly conceptual and judgement-based, so they are among the easier points to win if you learn the vocabulary and the logic. Each item set has four questions drawn from one vignette, and a clear grasp of biases and method differences lets you answer several quickly. The same ideas also help you in Portfolio Construction, Quantitative Methods and Ethics questions, so the effort pays off across the paper.

Backtesting and Simulation: topics in the order to study them

  1. 1Backtesting Basics and PurposeStart here to learn what a backtest is, what it needs (rules, data, a test period) and what it can and cannot prove.
  2. 2Backtesting Pitfalls and BiasesOnce you know how a backtest works, learn how it goes wrong: look-ahead, survivorship, data snooping, overfitting and unrealistic costs.
  3. 3Historical and Parametric SimulationThese are the simpler simulation methods, and comparing them sets up the contrast with Monte Carlo.
  4. 4Monte Carlo Simulation in Portfolio PlanningThis builds on parametric ideas by adding random draws, multiple periods and planning uses such as spending and retirement goals.
  5. 5Interpreting Results and LimitationsFinish by judging outputs, since this skill ties together every earlier topic and is what item sets test.

How to prepare Backtesting and Simulation

Treat this chapter as a skill in questioning results, not a set of formulas to memorise. Build it in layers and practise on vignettes.

  1. Read the chapter once for the big picture: write one sentence on what each method does and what it assumes.
  2. Build a bias table in your own words: for each bias, note what goes wrong, a typical vignette clue, and the usual fix such as out-of-sample testing or point-in-time data.
  3. Compare historical, parametric and Monte Carlo side by side on data source, assumptions, ability to capture tail events, and flexibility.
  4. Practise reading vignettes: underline the data used, the test period, the assumptions and any choices made after seeing results.
  5. Do item sets under time and for each question state the method, the flaw or assumption, and the conclusion before looking at options.
  6. Review wrong answers by asking whether you misread the clue or confused two similar terms, and add that to your bias table.
  7. Spend the last session on limitations: model risk, garbage in garbage out, parameter uncertainty and false precision.

Common mistakes in Backtesting and Simulation

  • Treating a strong backtest as proof that a strategy will work

    Fix: Remember a backtest is evidence, not proof. Check for bias, out-of-sample support and economic rationale.

  • Mixing up look-ahead bias and survivorship bias

    Fix: Ask: was future information used (look-ahead), or were failed entities left out (survivorship)?

  • Saying Monte Carlo is always better than historical simulation

    Fix: Monte Carlo depends on the assumed model and inputs. Historical simulation uses real observed data but is limited to past events. Judge by the vignette.

  • Ignoring choices made after seeing the results

    Fix: Look for rules or parameters tuned after viewing results. That points to data snooping or overfitting.

  • Overlooking input quality in simulation vignettes

    Fix: Check how returns, volatilities and correlations were estimated, and whether they suit the horizon and market conditions.

  • Forgetting trading costs and practical constraints

    Fix: When a vignette omits costs, liquidity or taxes, treat that as a reason performance may be overstated.

Last-day revision: Backtesting and Simulation

  • Backtesting applies a rule to past data to estimate how it would have performed.
  • Look-ahead bias uses information that was not available at the decision date.
  • Survivorship bias uses only entities that still exist, which overstates performance.
  • Data snooping means testing many rules and reporting the best, so the result may be luck.
  • Overfitting fits noise in the sample and tends to fail out of sample.
  • Out-of-sample testing and point-in-time data are the main defences against bias.
  • Historical simulation resamples actual past returns and assumes the past represents the future.
  • Parametric simulation assumes a distribution and estimated parameters, so wrong inputs mean wrong output.
  • Monte Carlo draws random values from a specified model and can handle complex, path-dependent problems.
  • Simulation output is only as good as its assumptions: garbage in, garbage out.
  • Results can show false precision, so treat them as ranges and probabilities, not guarantees.
  • Realistic backtests include transaction costs, taxes and market impact.

Backtesting and Simulation in other exams

The same ground in other exams, if you are preparing for more than one or want another angle on it.

Backtesting and Simulation: frequently asked questions

What is the difference between backtesting and simulation?

Backtesting tests a specific rule or strategy on past data to see how it would have done. Simulation generates many possible outcomes, from past data, an assumed distribution or random draws, to study the range of future results.

How do historical, parametric and Monte Carlo simulation differ?

Historical simulation resamples actual past returns. Parametric simulation assumes a distribution and uses estimated parameters. Monte Carlo draws random values from a specified model and can handle many periods and complex features.

Do I need to do calculations in this chapter?

Mostly the focus is on concepts, method choice and spotting flaws, so heavy calculation is less likely. Still, be comfortable reading outputs such as probabilities of reaching a goal or distributions of results from a vignette.

How do I spot bias in a backtest vignette?

Look at the data source, the dates, what information was available at each decision, which entities are included, and whether rules were adjusted after seeing results. Each clue points to a specific bias.

Why does the exam care about limitations of simulation?

Because analysts often present model output as certain. The exam tests whether you can see that results depend on assumptions, input estimates and model design, and communicate that honestly.