Backtesting and Simulation: formula sheet

Key formulas

Backtest purpose (rule)

Strategy rules + historical data → hypothetical performance → compare with benchmark or expectation

This is the core logic. Backtesting is a process, not a calculation.

Excess return versus benchmark

Active return = Backtested portfolio return − Benchmark return

Use the same period and the same return basis (for example, both net of costs).

Risk-model exceedance rate

Exceedance rate = Number of days actual loss exceeds VaR ÷ Number of days tested

Compare it with the stated tail probability. A 1% VaR should be exceeded on roughly 1% of days.

In-sample versus out-of-sample

Fit or tune on in-sample data; judge on out-of-sample data

Results from data used to design the rules are biased upward.

Look-ahead bias

Test uses information dated after the decision date

Fix with point-in-time, as-originally-reported data.

Survivorship bias

Sample = only entities that survived to the end date

Overstates average return and understates risk. Fix by including dead and delisted entities.

Data snooping

Many tests on the same data → some pass by chance

Fix with out-of-sample testing, fewer tests, and an economic rationale.

Overfitting

High in-sample fit, weak out-of-sample fit

A large gap between the two signals overfitting.

Net return

Net return = gross return − trading costs − other costs

Costs include commissions, spreads and market impact. Turnover raises their effect.

Parametric draw for a single asset (normal model)

Rₜ = μ + σ × z, where z is a standard normal random draw

μ is the assumed mean return and σ the assumed standard deviation. Used for one period.

Correlated multi-asset draws

Draws must reflect the covariance matrix (variances and covariances) of the assets

Independent draws would ignore correlation and overstate diversification.

Compounding simulated returns to a terminal value

Vₜ₊₁ = Vₜ × (1 + Rₜ)

Compounding simulated arithmetic returns period by period gives a terminal value for each path. If returns are modelled as continuously compounded and normally distributed (a lognormal price path), use Vₜ₊₁ = Vₜ × e^(rₜ) instead.

Probability estimate from simulation

P(event) ≈ number of paths where event occurs ÷ total number of paths

For example, probability of shortfall equals paths ending below the goal divided by all paths.

Probability of success

Success probability = number of trials meeting the goal ÷ total number of trials

Failure probability = 1 − success probability. Goal can be a target wealth or money lasting through the horizon.

Portfolio value roll-forward (one period)

Vₜ = (Vₜ₋₁ − withdrawal) × (1 + rₜ)

Timing of the withdrawal matters. Use the order the vignette states. rₜ is the random draw for that period.

Lognormal return draw

Vₜ = Vₜ₋₁ × e^rₜ, where rₜ is a continuously compounded return

Use when the vignette says continuously compounded returns are normally distributed. Prices cannot go below zero.

Standard error of a simulated mean

SE = s ÷ √N

s is the standard deviation of the trial results and N the number of trials. Quadrupling trials halves the error.

Out-of-sample test

Estimate on in-sample data, then evaluate on data not used in fitting

A large drop in performance out of sample signals overfitting.

Simulation standard error (rule of thumb)

Standard error of the estimated mean ≈ s ÷ √N

s is the sample standard deviation of outcomes, N is the number of trials. To halve the error you need about four times as many trials. This covers sampling error only, not model error.

Expected number of VaR exceptions

Expected exceptions = (1 − confidence level) × number of observations

Compare actual exceptions with this figure when backtesting VaR. Far more exceptions suggest risk is understated.

Quick revision

• Backtesting applies a rule to past data to estimate how it would have performed.
• Look-ahead bias uses information that was not available at the decision date.
• Survivorship bias uses only entities that still exist, which overstates performance.
• Data snooping means testing many rules and reporting the best, so the result may be luck.
• Overfitting fits noise in the sample and tends to fail out of sample.
• Out-of-sample testing and point-in-time data are the main defences against bias.
• Historical simulation resamples actual past returns and assumes the past represents the future.
• Parametric simulation assumes a distribution and estimated parameters, so wrong inputs mean wrong output.
• Monte Carlo draws random values from a specified model and can handle complex, path-dependent problems.
• Simulation output is only as good as its assumptions: garbage in, garbage out.
• Results can show false precision, so treat them as ranges and probabilities, not guarantees.
• Realistic backtests include transaction costs, taxes and market impact.

Common mistakes

• Treating a strong backtest as proof of future performance Fix: Remember it only shows hypothetical past results. The future can differ from the sample.
• Counting in-sample results as validation Fix: Look for words like optimized, tuned or selected. Real validation needs data not used to design the rules.
• Confusing look-ahead bias with survivorship bias. Fix: Look-ahead is about timing of information. Survivorship is about which entities are in the sample.
• Treating data snooping and overfitting as the same thing. Fix: Snooping is repeated testing and selecting the best. Overfitting is a model too closely tuned to noise. Match the vignette's wording.
• Saying historical simulation needs a distribution assumption Fix: Historical simulation is non-parametric. It uses the empirical data as it is.
• Assuming parametric simulation captures fat tails Fix: If the model is normal, tails are thin. Fat tails appear only if the distribution is chosen to include them.
• Treating the success probability as a guarantee. Fix: It is an estimate conditional on the assumed distributions. Say the result depends on inputs.
• Thinking more trials fix bad assumptions. Fix: More trials reduce sampling error only. They do not correct a wrong distribution or wrong parameters.
• Believing a high backtest return proves the strategy works. Fix: Ask whether it was tested out of sample and how many rules were tried to find it.
• Thinking more Monte Carlo trials fix a bad model. Fix: More trials only reduce random error. Wrong distributions or correlations remain wrong.

Exam tips

• Read the vignette for how the rules were chosen. Tuning on the same data is the usual trap.
• If an option says a backtest proves or guarantees future returns, eliminate it.
• For VaR backtests, compute exceedances ÷ days and compare with the stated tail probability.
• Check whether figures are gross or net of costs before comparing strategies.
• Keep backtesting (real past data) separate from simulation (generated scenarios).
• Read the vignette for timing words such as 'restated', 'final' and 'as of year-end'. They often signal look-ahead bias.
• Name the direction: nearly every bias here overstates return or understates risk.
• When asked for a remedy, tie it to the flaw: point-in-time data, including delisted entities, out-of-sample testing, or realistic costs.