Regression Diagnostics: formula sheet

Key formulas

Multiple regression model

Yi = β0 + β1X1i + β2X2i + ... + βkXki + εi

k regressors plus an intercept, so k + 1 coefficients.

OLS objective

minimise SSR = Σ (Yi − Ŷi)² = Σ ei²

OLS chooses the estimates that give the smallest sum of squared residuals.

Fitted value and residual

Ŷi = b0 + b1X1i + ... + bkXki; ei = Yi − Ŷi

Residuals sum to zero when the model has an intercept.

Degrees of freedom

df = n − k − 1

n observations, k slope coefficients. Used for t-tests and the standard error of regression.

Standard error of regression

SER = √(SSR ÷ (n − k − 1))

Estimate of the standard deviation of the error term.

Zero conditional mean

E(ε | X1, ..., Xk) = 0

Failure causes biased and inconsistent OLS estimates.

Homoskedasticity

Var(ε | X) = σ², a constant

Violation is heteroskedasticity.

No serial correlation

Cov(εi, εj) = 0 for i ≠ j

Violation is autocorrelation, common in time series.

Gauss-Markov / BLUE

Under the classical assumptions, OLS has the minimum variance among linear unbiased estimators

Does not require normal errors.

Homoskedasticity assumption

Var(εᵢ | X) = σ² for all i

Heteroskedasticity means Var(εᵢ | X) = σᵢ², which varies by observation.

Breusch-Pagan auxiliary regression

ε̂ᵢ² = α₀ + α₁X₁ + … + αₖXₖ + uᵢ

Regress squared residuals on the regressors. Null hypothesis: all α slopes = 0 (homoskedasticity).

Breusch-Pagan test statistic

LM = n × R² ~ χ²(k)

R² is from the auxiliary regression, n is the sample size, k is the number of regressors in it. Reject the null if LM exceeds the critical value.

White test

ε̂ᵢ² regressed on the Xs, their squares and cross-products; LM = n × R² ~ χ²(q)

q is the number of auxiliary regressors excluding the intercept. It detects nonlinear forms of heteroskedasticity.

Weighted least squares

Divide each observation by σᵢ, then run OLS

Needs a known or estimated variance pattern. Robust standard errors do not.

Variance Inflation Factor

VIF_j = 1 ÷ (1 − R_j²)

R_j² comes from regressing regressor j on all the other regressors (not on Y). VIF = 1 means no collinearity with the others.

Effect on slope variance

Var(β̂_j) = σ² ÷ [(1 − R_j²) × Σ(x_ij − x̄_j)²]

The standard error is inflated by √VIF compared with uncorrelated regressors. Standard form for multiple regression.

Standard error inflation

SE inflation factor = √VIF

A VIF of 9 means the standard error is 3 times as large as it would be with no collinearity.

Rule of thumb

VIF > 10 suggests serious multicollinearity

A common guideline, not a strict test. Some texts use lower cut-offs such as 5.

Two-regressor case

R_j² = r² (squared correlation between the two regressors)

With only two regressors, VIF = 1 ÷ (1 − r²) for both.

Classic symptom

High R² and significant F, but insignificant individual t-stats

Pairwise correlations can miss collinearity involving three or more variables, so VIF is more reliable.

Durbin-Watson statistic

DW = Σ(et − et−1)² ÷ Σet²

Numerator sums from t = 2 to n; denominator from t = 1 to n. et are OLS residuals.

DW and first-order autocorrelation

DW ≈ 2(1 − ρ)

Approximation for large samples. ρ = 0 gives DW ≈ 2; ρ near +1 gives DW near 0; ρ near −1 gives DW near 4.

Solving for ρ

ρ ≈ 1 − DW ÷ 2

Use to estimate the residual autocorrelation from a reported DW.

AR(1) error process

εt = ρεt−1 + ut

ut is white noise. |ρ| < 1 for stationary errors. Serial correlation exists if ρ ≠ 0.

DW decision rule

DW < dL: reject H0 (positive autocorrelation); DW > dU: do not reject; dL ≤ DW ≤ dU: inconclusive

H0: ρ = 0. dL and dU come from tables based on n and the number of regressors. For negative autocorrelation, compare 4 − DW with the same bounds.

OVB in a two-variable case

True: Y = β0 + β1·X1 + β2·X2 + ε. Omit X2: E[β1_hat] = β1 + β2 · δ, where δ = Cov(X1, X2) ÷ Var(X1)

Bias = β2 · δ. δ is the slope from regressing X2 on X1. Sign of bias = sign of β2 times sign of the correlation.

Conditions for OVB

β2 ≠ 0 AND Corr(X1, X2) ≠ 0

Both are needed. If either is zero, the slope on X1 is unbiased.

Effect of an irrelevant variable

Bias = 0; Var(β1_hat) increases when the extra variable is correlated with X1

Unbiased but less efficient.

Adjusted R²

Adj R² = 1 − [(n − 1) ÷ (n − k − 1)] × (1 − R²)

k = number of regressors excluding the intercept. It can fall when a useless variable is added.

Information criteria

AIC = ln(SSR ÷ n) + 2k ÷ n; BIC = ln(SSR ÷ n) + k·ln(n) ÷ n

Lower is better. BIC penalty is larger than AIC when n ≥ 8.

Residual

eᵢ = Yᵢ − Ŷᵢ

Actual minus fitted. OLS residuals sum to zero when the model has an intercept.

Standardized residual (approximate)

eᵢ ÷ s, where s = √(SSR ÷ (n − k − 1))

s is the standard error of the regression. Absolute values above about 2 to 3 are often flagged as outliers. Exact versions also adjust for leverage.

Leverage average

Average hᵢᵢ = (k + 1) ÷ n

k is the number of explanatory variables. Leverages sum to k + 1. Values well above the average, such as more than twice it, are often flagged.

Leverage in simple regression

hᵢᵢ = 1/n + (Xᵢ − X̄)² ÷ Σ(Xⱼ − X̄)²

Grows as Xᵢ moves away from the mean of X.

Cook's distance

Dᵢ = (eᵢ² ÷ ((k + 1) × s²)) × (hᵢᵢ ÷ (1 − hᵢᵢ)²)

Large when both the residual and the leverage are large. Common flags: Dᵢ > 1, or Dᵢ > 4 ÷ n.

Residual-plot reading rule

Pattern in mean → misspecification; pattern in spread → heteroskedasticity; pattern in time order → serial correlation

Use this to link a picture to the problem.

Quick revision

• Heteroskedasticity means non-constant error variance; OLS coefficients stay unbiased but standard errors are unreliable.
• Robust (White) standard errors are the usual fix for heteroskedasticity.
• Common tests for heteroskedasticity include Breusch-Pagan and White.
• Perfect multicollinearity makes OLS impossible; imperfect multicollinearity inflates standard errors.
• Classic multicollinearity symptom: high R² and a significant F-test but insignificant individual t-statistics.
• A high variance inflation factor (VIF) or high pairwise correlations point to multicollinearity.
• Positive serial correlation typically makes standard errors too small and t-statistics too large.
• Durbin-Watson is near 2 with no first-order autocorrelation, below 2 for positive, above 2 for negative.
• Omitted variable bias needs the omitted variable to be both correlated with an included regressor and a determinant of the dependent variable.
• Adding regressors never lowers R², so use adjusted R² to compare models.
• Outliers can have a large influence on coefficients; investigate before deleting.
• Residual plots should show no pattern; a funnel suggests heteroskedasticity, a wave suggests autocorrelation.

Common mistakes

• Saying heteroskedasticity makes OLS coefficients biased. Fix: Remember that heteroskedasticity and serial correlation leave estimates unbiased but make them inefficient and the standard errors incorrect.
• Treating normality of errors as a Gauss-Markov requirement. Fix: Gauss-Markov needs only linearity, zero conditional mean, no perfect multicollinearity, homoskedasticity and no serial correlation. Normality is for exact small-sample inference.
• Saying heteroskedasticity makes OLS coefficients biased. Fix: Remember that only the standard errors and efficiency suffer. Bias comes from omitted variables or correlation between errors and regressors.
• Assuming standard errors are always too small. Fix: Say the standard errors are unreliable. They are often too small, but the direction depends on the pattern.
• Saying multicollinearity biases the OLS coefficients. Fix: Remember imperfect multicollinearity leaves OLS unbiased and consistent. It only inflates variances.
• Computing VIF using R² from the regression of Y on X. Fix: R_j² comes from regressing X_j on the other regressors. Y is not involved.
• Saying serial correlation makes OLS coefficients biased. Fix: With strictly exogenous regressors, OLS slopes stay unbiased and consistent. The standard errors are what fail.
• Treating DW = 0 as no autocorrelation. Fix: No autocorrelation gives DW ≈ 2. DW near 0 means strong positive autocorrelation; near 4 means strong negative.
• Saying any omitted variable causes bias. Fix: Always check both conditions: the variable affects Y and correlates with an included regressor.
• Claiming irrelevant variables bias the coefficients. Fix: Irrelevant variables leave OLS unbiased and consistent; they only inflate variance.

Exam tips

• Expect conceptual questions that ask which assumption violation causes bias and which only hurts efficiency.
• Memorise that df = n − k − 1 and compute it first in any test-statistic question.
• When a coefficient is asked to be interpreted, include units and the holding-constant phrase to rule out wrong options.
• Read BLUE word by word: best (minimum variance), linear, unbiased, estimator.
• Watch for the distractor that adds normality to Gauss-Markov. It is not required.
• Expect conceptual questions on consequences: unbiased coefficients, wrong standard errors, inflated t-statistics.
• Know the Breusch-Pagan and White tests by their auxiliary regression and n × R² statistic.
• When asked for the simplest correction, choose robust standard errors.