Core concepts of time series models: formula sheet

Key formulas

Mean function

μt = E[Xt]

Weak stationarity requires μt = μ, the same for all t.

Autocovariance function

γk = Cov(Xt, Xt+k) = E[(Xt − μ)(Xt+k − μ)]

For a weakly stationary process this depends only on the lag k. γ0 = Var(Xt).

Autocorrelation function

ρk = γk ÷ γ0

ρ0 = 1, |ρk| ≤ 1 and ρ−k = ρk.

Weak stationarity conditions

E[Xt] = μ for all t; Var(Xt) < ∞; Cov(Xt, Xt+k) = γk for all t

All three must hold. Failing any one means the series is not weakly stationary.

Strict stationarity

(Xt1, ..., Xtn) has the same joint distribution as (Xt1+k, ..., Xtn+k) for all n, t1, ..., tn, k

Strict plus finite variance implies weak. Weak implies strict only for Gaussian processes.

Sample autocovariance and ACF

ck = (1 ÷ n) Σ (xt − x̄)(xt+k − x̄), sum over t = 1 to n − k; rk = ck ÷ c0

Used to estimate γk and ρk from data. Divide by n, not n − k, in the standard definition.

White noise definition

E[et] = 0, Var(et) = σ², Cov(et, es) = 0 for t ≠ s

Uncorrelated is enough. Independence is a stronger, separate assumption.

White noise autocorrelation

ρk = 1 for k = 0, and ρk = 0 for k ≥ 1

Sample autocorrelations of white noise should lie within about ±2/√n.

Random walk

Xt = Xt-1 + et, so Xt = X0 + e1 + ... + et

With X0 fixed (often 0).

Random walk mean and variance

E[Xt] = X0, Var(Xt) = tσ²

Assumes X0 is a constant. Variance grows with t, so not stationary.

Random walk autocovariance

Cov(Xt, Xt+k) = tσ² for k ≥ 0

Depends on t, not just the lag k.

Random walk with drift

Xt = Xt-1 + μ + et, E[Xt] = X0 + μt, Var(Xt) = tσ²

Drift changes the mean only. The variance is unchanged.

Differencing

∇Xt = Xt - Xt-1 = μ + et

Result is stationary: white noise with mean μ.

AR(p) model

X_t = μ + α₁(X_{t-1} − μ) + … + αₚ(X_{t-p} − μ) + e_t

e_t is white noise with mean 0 and variance σ². Written with the mean μ, as in IAI notation.

Stationarity condition

All roots of 1 − α₁z − α₂z² − … − αₚzᵖ = 0 satisfy |z| > 1

For AR(1): |α| < 1. Roots may be complex; use the modulus.

AR(1) ACF

ρ_k = αᵏ for k ≥ 0

Decays geometrically. Alternates in sign if α < 0.

AR(1) variance

γ₀ = σ² ÷ (1 − α²)

Needs |α| < 1.

Yule-Walker recursion for ACF

ρ_k = α₁ρ_{k-1} + α₂ρ_{k-2} + … + αₚρ_{k-p} for k ≥ 1

Use ρ₀ = 1 and ρ_{-k} = ρ_k.

AR(2) Yule-Walker, first two lags

ρ₁ = α₁ + α₂ρ₁ and ρ₂ = α₁ρ₁ + α₂

So ρ₁ = α₁ ÷ (1 − α₂).

AR(2) stationarity triangle

α₁ + α₂ < 1, α₂ − α₁ < 1, |α₂| < 1

All three must hold. This is equivalent to the root condition.

Variance from autocovariances

γ₀ = σ² ÷ (1 − α₁ρ₁ − … − αₚρₚ)

Gives the variance once the ρ values are known.

PACF of AR(p)

φ_kk ≠ 0 for k ≤ p, φ_kk = 0 for k > p; φ₁₁ = ρ₁ and φ_pp = α_p

Cut-off at lag p identifies the order.

MA(q) definition

X_t = μ + e_t + θ₁e_{t-1} + ... + θ_q e_{t-q}, with e_t white noise with variance σ²

Always weakly stationary. Mean is μ.

Variance of MA(q)

γ₀ = σ²(1 + θ₁² + ... + θ_q²)

Shocks are uncorrelated, so squared coefficients add.

Autocovariance of MA(q)

γ_k = σ²(θ_k + θ₁θ_{k+1} + ... + θ_{q-k}θ_q) for 1 ≤ k ≤ q, with θ₀ = 1; γ_k = 0 for k > q

Sum the products of coefficients that line up on the same shock.

MA(1) ACF

ρ₁ = θ ÷ (1 + θ²); ρ_k = 0 for k ≥ 2

|ρ₁| ≤ 0.5 always. The maximum is at θ = ±1.

MA(2) ACF

ρ₁ = θ₁(1 + θ₂) ÷ (1 + θ₁² + θ₂²); ρ₂ = θ₂ ÷ (1 + θ₁² + θ₂²); ρ_k = 0 for k ≥ 3

Check lag 1 numerator carefully: θ₁ + θ₁θ₂.

Invertibility condition

All roots of 1 + θ₁z + ... + θ_q z^q = 0 satisfy |z| > 1

Equivalent to all roots outside the unit circle.

MA(1) invertibility

|θ| < 1

Root of 1 + θz = 0 is z = −1/θ.

MA(2) invertibility region

θ₂ + θ₁ > −1, θ₂ − θ₁ > −1, |θ₂| < 1

This is the triangle region for the parameters. Using the roots directly also works.

Backward shift operator

B X_t = X_{t-1}; B^k X_t = X_{t-k}

B acts on the time index only. B applied to a constant gives the same constant.

ARMA(p,q) model

X_t = μ + φ₁(X_{t-1} − μ) + … + φ_p(X_{t-p} − μ) + e_t + θ₁e_{t-1} + … + θ_q e_{t-q}

e_t is white noise with variance σ². Check which sign convention the question uses for θ.

ARMA in operator form

φ(B)(X_t − μ) = θ(B) e_t, with φ(B) = 1 − φ₁B − … − φ_p B^p and θ(B) = 1 + θ₁B + … + θ_q B^q

Move all X terms to the left and all e terms to the right.

Stationarity condition

All roots of φ(z) = 0 satisfy |z| > 1

Only the AR polynomial matters. Equivalent: roots of the reversed polynomial lie inside the unit circle.

Invertibility condition

All roots of θ(z) = 0 satisfy |z| > 1

Only the MA polynomial matters.

Difference operator

∇ = 1 − B; ∇X_t = X_t − X_{t-1}; ∇^d = (1 − B)^d

∇² X_t = X_t − 2X_{t-1} + X_{t-2}. It is not X_t − X_{t-2}.

ARIMA(p,d,q) model

φ(B)(1 − B)^d X_t = θ(B) e_t

Here the differenced series Y_t = ∇^d X_t is a stationary, invertible ARMA(p,q). The ARIMA process itself is not stationary when d ≥ 1.

AR(1) stationarity

X_t = φX_{t-1} + e_t is stationary if |φ| < 1

φ(z) = 1 − φz has root z = 1/φ.

MA(1) invertibility

X_t = e_t + θe_{t-1} is invertible if |θ| < 1

θ(z) = 1 + θz has root z = −1/θ.

Identification pattern for AR(p)

PACF cuts off after lag p; ACF decays

Sample values are inside ±2/√n roughly when the true value is zero, where n is the sample size.

Identification pattern for MA(q)

ACF cuts off after lag q; PACF decays

For ARMA both decay, so compare candidate models using diagnostics and AIC.

Approximate significance band

±1.96/√n (often quoted as ±2/√n)

A sample autocorrelation outside the band is taken as significantly different from 0.

AR(1) model

X_t − μ = α(X_{t−1} − μ) + e_t, |α| < 1

e_t is white noise with variance σ².

AR(1) k-step forecast

x̂_t(k) = μ + α^k (x_t − μ)

Replace α and μ by estimates when fitting from data.

AR(1) k-step forecast error variance

σ² (1 − α^{2k}) ÷ (1 − α²)

Tends to σ² ÷ (1 − α²), the process variance, as k grows.

Yule-Walker for AR(1)

α̂ = r₁ (sample lag-1 autocorrelation)

For AR(1), ρ₁ = α.

Information criterion

AIC = −2 ln L + 2 × (number of parameters)

Smaller is better. Compare models fitted to the same data.

Differencing

∇X_t = X_t − X_{t−1}

ARIMA(p, d, q) means ∇^d X_t is a stationary ARMA(p, q).

Quick revision

• Weak stationarity: constant mean, constant finite variance, and covariance that depends only on the lag.
• Autocorrelation at lag k: ρk = γk ÷ γ0, and ρ0 = 1.
• White noise: uncorrelated terms with zero mean and constant variance.
• Random walk: Xt = Xt-1 + et is non-stationary, but its first difference is white noise.
• AR(p) is stationary when all roots of its characteristic polynomial lie outside the unit circle.
• AR(1): Xt = μ + α(Xt-1 − μ) + et is stationary if |α| < 1, with ρk = α^k.
• MA(q) is always stationary and its ACF is zero beyond lag q.
• MA(q) is invertible when all roots of its MA polynomial lie outside the unit circle.
• AR has a PACF that cuts off after lag p; MA has an ACF that cuts off after lag q.
• ARIMA(p,d,q) means the series differenced d times follows an ARMA(p,q) model.
• Stationary forecasts tend to the mean as the horizon grows; random-walk forecasts do not.
• Check residuals: they should look like white noise, with no remaining ACF pattern.

Common mistakes

• Checking only that the mean is constant and then declaring the process stationary. Fix: Always test the variance and the autocovariance too. All three conditions must hold.
• Saying weak stationarity implies strict stationarity. Fix: Strict plus finite variance implies weak. The reverse holds only in special cases such as Gaussian processes.
• Saying a random walk has variance σ² because the shocks have variance σ². Fix: Xt is a sum of t shocks. Variance is tσ². Only the first difference has variance σ².
• Adding drift into the variance, for example Var(Xt) = tσ² + μt. Fix: Drift is a constant. It shifts the mean and has zero variance.
• Using |α| > 1 or the wrong side of the root condition. Fix: For AR(1), the root of 1 − αz = 0 is z = 1/α. It must lie outside the unit circle, so |α| < 1. Check which polynomial you are using.
• Checking only α₁ + α₂ < 1 for AR(2). Fix: Check all three: α₁ + α₂ < 1, α₂ − α₁ < 1 and |α₂| < 1.
• Saying an MA process needs a stationarity condition on θ. Fix: An MA process is always stationary. The root condition on an MA polynomial is for invertibility, not stationarity.
• Forgetting the 1 in the variance formula. Fix: Write γ₀ = σ²(1 + θ₁² + ... + θ_q²) every time, with θ₀ = 1.
• Declaring an ARIMA(p,d,q) process with d ≥ 1 to be stationary because the ARMA part satisfies the root conditions. Fix: Always say: the differenced series ∇^d X_t is stationary, but X_t itself is not stationary when d ≥ 1.
• Testing stationarity using the MA polynomial, or invertibility using the AR polynomial. Fix: Remember: AR part gives stationarity, MA part gives invertibility. Write this beside your working.

Exam tips

• Start every stationarity answer by computing E[Xt]. It is quick and often settles the question.
• Show the covariance working line by line. Marks are given for method, not only for the verdict.
• State which noise assumptions you use, for example that et is white noise with mean 0 and variance σ².
• If a question mentions a Gaussian process, say that weak stationarity then implies strict stationarity.
• In computer-based papers, plot the series and use the acf() function in R to show the sample ACF, then explain what a slow decay means.
• Always state the stationarity conclusion with its reason. Marks are usually given for naming which condition fails.
• In written answers, show the expansion Xt = X0 + μt + e1 + ... + et before taking moments.
• For MCQs, test the options quickly: mean X0 + μt and variance tσ² are the two values most often examined.