Applications of time series models: formula sheet

Key formulas

Additive decomposition

X_t = m_t + s_t + Z_t

m_t is trend, s_t is seasonal (repeats every d periods, so s_t+d = s_t), Z_t is random.

Multiplicative decomposition

X_t = m_t × s_t × Z_t

Taking logs turns it into an additive model.

Weak stationarity conditions

E[X_t] = μ for all t; Var(X_t) < ∞; Cov(X_t, X_t+k) = γ_k depends only on k

All three must hold. Mean cannot depend on t.

Autocovariance function

γ_k = Cov(X_t, X_t+k) = E[(X_t − μ)(X_t+k − μ)]

γ_0 = Var(X_t). Also γ_−k = γ_k.

Autocorrelation function

ρ_k = γ_k ÷ γ_0

ρ_0 = 1 and |ρ_k| ≤ 1.

Sample autocovariance

c_k = (1/n) Σ (x_t − x̄)(x_t+k − x̄), sum over t = 1 to n − k

Divide by n, as in the standard sample estimator. Sample ACF is r_k = c_k ÷ c_0.

Differencing

∇X_t = X_t − X_t−1; seasonal: ∇_d X_t = X_t − X_t−d

First differences remove a linear trend; seasonal differences remove a seasonal pattern of period d.

AR(p) definition

X_t = μ + α₁(X_{t-1} − μ) + ... + α_p(X_{t-p} − μ) + e_t

Using the backward shift operator B: φ(B)(X_t − μ) = e_t, with φ(B) = 1 − α₁B − ... − α_pB^p.

MA(q) definition

X_t = μ + e_t + β₁e_{t-1} + ... + β_q e_{t-q}

Written as X_t − μ = θ(B)e_t, with θ(B) = 1 + β₁B + ... + β_qB^q.

Stationarity condition

All roots of 1 − α₁z − ... − α_p z^p = 0 satisfy |z| > 1

Applies to the AR part of AR and ARMA. MA processes are always stationary.

Invertibility condition

All roots of 1 + β₁z + ... + β_q z^q = 0 satisfy |z| > 1

Applies to the MA part of MA and ARMA.

AR(1) stationarity and ACF

|α| < 1; ρ_k = α^k

Variance γ₀ = σ² ÷ (1 − α²).

AR(2) stationarity region

α₁ + α₂ < 1, α₂ − α₁ < 1, |α₂| < 1

All three must hold. This is the triangle region.

AR(2) Yule-Walker equations

ρ₁ = α₁ ÷ (1 − α₂); ρ_k = α₁ρ_{k-1} + α₂ρ_{k-2} for k ≥ 2

Use these to find ACF values recursively.

MA(1) ACF

ρ₁ = β ÷ (1 + β²); ρ_k = 0 for k ≥ 2

Variance γ₀ = σ²(1 + β²). Invertible if |β| < 1.

MA(q) variance

γ₀ = σ²(1 + β₁² + ... + β_q²)

Autocovariance at lag k is σ²(β_k + β₁β_{k+1} + ... + β_{q-k}β_q), with β₀ = 1.

ARMA(1,1) ACF

ρ₁ = (1 + αβ)(α + β) ÷ (1 + 2αβ + β²); ρ_k = α ρ_{k-1} for k ≥ 2

Model: X_t = αX_{t-1} + e_t + βe_{t-1}, with |α| < 1 and |β| < 1.

ACF and PACF patterns

AR(p): ACF decays, PACF cuts off after p. MA(q): ACF cuts off after q, PACF decays. ARMA: both decay.

The main identification rule.

Difference operator

∇Xₜ = Xₜ − Xₜ₋₁ = (1 − B)Xₜ

B is the backshift operator: BXₜ = Xₜ₋₁.

Higher-order difference

∇²Xₜ = Xₜ − 2Xₜ₋₁ + Xₜ₋₂ = (1 − B)²Xₜ

This is the difference of the first differences, not Xₜ − Xₜ₋₂.

ARIMA(p,d,q) definition

φ(B)(1 − B)ᵈ Xₜ = θ(B) eₜ

φ(B) = 1 − φ₁B − … − φₚBᵖ and θ(B) = 1 + θ₁B + … + θ_qB^q. eₜ is white noise. Conventions for signs of θ differ; state yours.

Integrated process

Xₜ ~ I(d) if ∇ᵈXₜ is stationary and ∇ᵈ⁻¹Xₜ is not

d is the smallest number of differences giving stationarity.

Seasonal difference

∇ₛXₜ = Xₜ − Xₜ₋ₛ = (1 − Bˢ)Xₜ

s is the season length, such as 12 for monthly data.

Seasonal ARIMA form

φ(B) Φ(Bˢ) (1 − B)ᵈ (1 − Bˢ)ᴰ Xₜ = θ(B) Θ(Bˢ) eₜ

Capital letters are the seasonal AR and MA polynomials in Bˢ.

Random walk as ARIMA

Xₜ = Xₜ₋₁ + eₜ is ARIMA(0,1,0)

With a constant added, it is a random walk with drift.

Yule-Walker equations for AR(p)

ρk = α1ρ(k−1) + α2ρ(k−2) + … + αpρ(k−p), for k = 1, 2, …, p

Here ρk is the autocorrelation at lag k and ρ0 = 1. Replace ρk with sample values to estimate α1, …, αp.

Yule-Walker for AR(1)

α̂1 = ρ̂1

The estimate of the AR(1) coefficient is the lag-1 sample autocorrelation.

Yule-Walker for AR(2)

ρ1 = α1 + α2ρ1 ; ρ2 = α1ρ1 + α2

Solve: α2 = (ρ2 − ρ1²) ÷ (1 − ρ1²), then α1 = ρ1(1 − α2).

Residual variance of AR(p)

σ² = γ0(1 − α1ρ1 − α2ρ2 − … − αpρp)

γ0 is the variance of the process. For AR(1) this gives σ² = γ0(1 − α1²).

Sample autocorrelation

ρ̂k = Σ(xt − x̄)(x(t+k) − x̄) ÷ Σ(xt − x̄)²

Numerator sums over t = 1 to n−k; denominator over t = 1 to n.

Approximate significance band

±2 ÷ √n

For white noise, sample ACF and PACF values are approximately N(0, 1/n). Values outside the band suggest a real correlation.

Ljung-Box portmanteau statistic

Q = n(n+2) Σ ρ̂k² ÷ (n−k), k = 1 to m

Compare with a chi-squared distribution. Degrees of freedom are m minus the number of fitted ARMA parameters (p + q). Large Q means the residuals are not white noise.

AIC

AIC = −2 ln L + 2k

k is the number of estimated parameters, L the maximised likelihood. Lower is better.

BIC

BIC = −2 ln L + k ln n

Penalises parameters more heavily than AIC when n ≥ 8, so it tends to choose smaller models.

Identification pattern

AR(p): PACF cuts off after p. MA(q): ACF cuts off after q.

ARMA(p, q): both decay, so use trial models and diagnostics.

Best forecast

x̂(n, k) = E[X(n+k) | X(n), X(n−1), ...]

Replace future e by 0, future X by forecasts, and keep past X and past e as known values.

AR(1) k-step forecast

x̂(n, k) = μ + α^k (x(n) − μ)

For X(t) − μ = α(X(t−1) − μ) + e(t). It tends to μ as k grows if |α| < 1.

MA(q) forecast

x̂(n, k) = μ for k > q

For k ≤ q, use the known past e terms with the correct θ coefficients, and set future e to 0.

MA(∞) form and ψ weights

X(t) = μ + Σ ψ(j) e(t−j), with ψ(0) = 1

For ARMA(1,1) with parameters α and θ: ψ(1) = α + θ and ψ(j) = α ψ(j−1) for j ≥ 2.

k-step forecast error

e(n, k) = X(n+k) − x̂(n, k) = Σ (j = 0 to k−1) ψ(j) e(n+k−j)

It has mean zero for the best forecast.

Forecast error variance

Var[e(n, k)] = σ² Σ (j = 0 to k−1) ψ(j)²

For AR(1): σ²(1 − α^(2k)) ÷ (1 − α²). For a 1-step forecast it equals σ².

ARIMA(p,1,q) forecasting

Forecast the differenced series Y(t) = X(t) − X(t−1), then x̂(n, k) = x(n) + Σ (j = 1 to k) ŷ(n, j)

For a random walk, x̂(n, k) = x(n) and the error variance is kσ².

Exponential smoothing

m̂(t) = α x(t) + (1 − α) m̂(t−1), and x̂(n, k) = m̂(n) for all k

Equivalent weights on past observations are α(1 − α)^j. A larger α reacts faster.

Integrated of order d

X(t) is I(d) if ∇^d X(t) is stationary and ∇^(d−1) X(t) is not

∇X(t) = X(t) − X(t−1). A random walk is I(1).

Cointegration

X(t), Y(t) both I(1), and Z(t) = X(t) − αY(t) is I(0) for some α

The vector (1, −α) is the cointegrating vector. A common test approach is to check whether the residuals of a regression are stationary.

VAR(1), two series

X(t) = a₁ + b₁₁X(t−1) + b₁₂Y(t−1) + e₁(t); Y(t) = a₂ + b₂₁X(t−1) + b₂₂Y(t−1) + e₂(t)

Errors may be correlated across equations. Stationarity requires the system to satisfy conditions on the coefficient matrix.

ARCH(1)

ε(t) = σ(t)·z(t), σ²(t) = α₀ + α₁ε²(t−1)

z(t) is white noise with mean 0 and variance 1. Need α₀ > 0 and α₁ ≥ 0. Stationary variance = α₀ ÷ (1 − α₁) when α₁ < 1.

GARCH(1,1)

σ²(t) = α₀ + α₁ε²(t−1) + β σ²(t−1)

Need α₀ > 0, α₁ ≥ 0, β ≥ 0. Finite stationary variance if α₁ + β < 1, equal to α₀ ÷ (1 − α₁ − β).

Wilkie inflation (structure)

ΔQ-type series: ln-inflation I(t) = μ + a·(I(t−1) − μ) + σ·z(t)

Inflation is an AR(1) process around a mean μ. Other series in the cascade depend on it. Exact parameter values are not memorised.

Quick revision

• A weakly stationary series has constant mean, constant variance and autocovariance that depends only on the lag.
• ACF at lag k is γk ÷ γ0, and ρ0 = 1.
• AR(p) is stationary when all roots of its characteristic polynomial lie outside the unit circle.
• MA(q) is always stationary, and it is invertible when the roots of its MA polynomial lie outside the unit circle.
• For AR(p), the PACF cuts off after lag p and the ACF decays.
• For MA(q), the ACF cuts off after lag q and the PACF decays.
• ARIMA(p,d,q) means the series differenced d times is ARMA(p,q).
• Box-Jenkins steps: identify, estimate, check the residuals, then forecast.
• Good residuals look like white noise, with no significant autocorrelation.
• In forecasts, future error terms are replaced by their expected value of zero.
• Forecast variance grows as you forecast further ahead.
• Differencing removes a trend, and seasonal differencing removes a seasonal pattern.

Common mistakes

• Saying a process is stationary because the mean is constant. Fix: Always check variance and autocovariance as well. A constant mean alone is not enough.
• Claiming weak stationarity implies strict stationarity. Fix: Only strict (with finite variance) implies weak. The converse holds in general only for Gaussian processes.
• Saying the roots must lie inside the unit circle. Fix: Write the polynomial as 1 − α₁z − ... and require |z| > 1. Check with AR(1): root z = 1/α, so |α| < 1 is the same as |z| > 1.
• Testing only α₁ and α₂ individually for AR(2). Fix: Use all three conditions: α₁ + α₂ < 1, α₂ − α₁ < 1 and |α₂| < 1.
• Treating the ARIMA series itself as stationary and applying ARMA ACF formulas to Xₜ. Fix: Only ∇ᵈXₜ is stationary. Apply ACF and stationarity results to the differenced series.
• Calculating the second difference as Xₜ − Xₜ₋₂. Fix: ∇²Xₜ = ∇(∇Xₜ) = Xₜ − 2Xₜ₋₁ + Xₜ₋₂. A lag-2 difference is a different operation.
• Reading the ACF cut-off to choose an AR order Fix: Memorise: PACF cuts off for AR, ACF cuts off for MA. The PACF at lag p is the last coefficient of the fitted AR(p).
• Fitting an ARMA model to a non-stationary series Fix: Always look at the plot and a slowly decaying ACF first. Difference, then identify the order of the differenced series.
• Forgetting to subtract the mean before applying the AR decay factor. Fix: Always use μ + α^k (x(n) − μ). The decay applies to the deviation from the mean, not to the level.
• Replacing a past noise term e(n) by zero in an MA or ARMA forecast. Fix: Only e(n+1), e(n+2), ... become zero. Terms up to time n are known and must be kept.

Exam tips

• Write the three conditions for weak stationarity explicitly and tick them off. Examiners award marks for each condition.
• In covariance questions, list the terms of each expression and match only the common independent terms. This avoids long expansions.
• When asked to comment on a sample ACF plot, mention slow decay, seasonal spikes at multiples of d, and the ±2/√n guide lines.
• State assumptions, such as Gaussian errors, when you move from weak to strict stationarity.
• In computer-based paper work, plot the series and ACF first and comment before fitting any model.
• Always state both conditions separately for ARMA: stationarity from the AR part, invertibility from the MA part.
• Show the three AR(2) inequalities in full. Marks are usually given for each check.
• Write the ACF working in standard notation: ρ_k, γ_k, with the recursion shown, not only the numeric result.