Regression with Multiple Explanatory Variables: formula sheet

Key formulas

Multiple regression model

Yi = β0 + β1X1i + β2X2i + ... + βkXki + εi

k regressors, k + 1 coefficients including the intercept.

Fitted value and residual

Ŷi = b0 + b1X1i + ... + bkXki; ei = Yi − Ŷi

OLS minimises Σei².

Interpretation of slope

ΔŶ = bj × ΔXj, other X held constant

Use this to predict the change in Y from a given change in one regressor.

Zero conditional mean

E(εi | X1i, ..., Xki) = 0

Key assumption for unbiasedness. Violated by omitted variables correlated with regressors.

Degrees of freedom

df = n − k − 1

Used for t tests of coefficients. n is observations, k is number of slope coefficients.

Standard error of regression

SER = √[SSR ÷ (n − k − 1)]

Estimate of the standard deviation of the error.

Perfect multicollinearity

Xj = c0 + c1X1 + ... (exact linear relation among regressors)

OLS cannot be computed. Classic case is the dummy variable trap.

t-statistic for a coefficient

t = (β̂ − β₀) ÷ SE(β̂)

β₀ is the hypothesized value. It is 0 in the standard significance test.

Degrees of freedom

df = n − k − 1

k is the number of slope coefficients. The intercept uses one more degree of freedom.

Confidence interval for a coefficient

β̂ ± t_critical × SE(β̂)

Use the two-sided critical value for the chosen confidence level, e.g. 5% total in both tails for 95%.

Decision rule (two-sided)

Reject H0 if |t| > t_critical, or if p-value < α

The two rules always agree.

Approximate large-sample critical values

90%: 1.645; 95%: 1.96; 99%: 2.576

These are normal values, a good approximation to t when df is large. For small df, use the t table given.

F-statistic (SSR form)

F = [(SSR_restricted − SSR_unrestricted) ÷ q] ÷ [SSR_unrestricted ÷ (n − k − 1)]

q = number of restrictions; k = slope coefficients in the unrestricted model; n = observations.

F-statistic (R² form)

F = [(R²_unrestricted − R²_restricted) ÷ q] ÷ [(1 − R²_unrestricted) ÷ (n − k − 1)]

Valid only when both models have the same dependent variable, so TSS is the same.

Overall significance F-test

F = (ESS ÷ k) ÷ (SSR ÷ (n − k − 1))

Null: all slope coefficients equal zero. Degrees of freedom (k, n − k − 1).

Degrees of freedom

numerator = q; denominator = n − k − 1

Denominator uses the unrestricted model.

Decision rule

Reject H0 if F > F critical (q, n − k − 1)

One-tailed, upper tail only.

Single restriction link

F = t² when q = 1

A one-restriction F-test gives the same conclusion as a two-sided t-test.

Sum of squares decomposition

TSS = ESS + SSR

TSS = Σ(Yᵢ − Ȳ)², ESS = Σ(Ŷᵢ − Ȳ)², SSR = Σ(Yᵢ − Ŷᵢ)². Holds for OLS with an intercept.

R-squared

R² = ESS ÷ TSS = 1 − SSR ÷ TSS

Share of variation in Y explained by the model.

Adjusted R-squared

Adjusted R² = 1 − [SSR ÷ (n − k − 1)] ÷ [TSS ÷ (n − 1)]

n = observations, k = number of explanatory variables (excluding the intercept).

Adjusted R-squared from R-squared

Adjusted R² = 1 − (1 − R²) × (n − 1) ÷ (n − k − 1)

The form you will use most often in the exam.

Standard error of regression

SER = √[SSR ÷ (n − k − 1)]

Uses the same degrees of freedom as adjusted R².

Link to F-statistic

F = (ESS ÷ k) ÷ [SSR ÷ (n − k − 1)]

Tests whether all slope coefficients are jointly zero.

Variance inflation factor

VIFⱼ = 1 ÷ (1 − Rⱼ²)

Rⱼ² comes from regressing Xⱼ on all the other regressors. VIF = 1 means no collinearity. A common rule of thumb flags VIF above 10 (some use 5). It is a guideline, not a strict test.

Variance of a slope coefficient (two regressors)

Var(β̂₁) = σ² ÷ [Σ(X₁ − X̄₁)² × (1 − r₁₂²)]

As the correlation r₁₂ between regressors approaches 1, the variance grows without bound.

Standard error inflation

SE with collinearity = SE without × √VIF

If VIF = 9, the standard error is 3 times larger than it would be with uncorrelated regressors.

t-statistic

t = (β̂ − β₀) ÷ SE(β̂)

Larger SE gives smaller t, so true effects look insignificant.

Perfect multicollinearity condition

Rⱼ² = 1, so VIFⱼ is undefined (infinite)

One regressor is an exact linear combination of others. OLS has no unique solution.

Homoskedasticity assumption

Var(ε | X1, ..., Xk) = σ² (constant)

Heteroskedasticity means this variance depends on X.

Breusch-Pagan LM statistic

LM = n × R² ~ χ²(k)

R² comes from regressing squared residuals on the k regressors. Null: homoskedasticity. A large LM rejects the null.

White test

LM = n × R² ~ χ²(q)

The auxiliary regression uses regressors, their squares and cross-products. q is the number of auxiliary regressors, excluding the intercept.

Omitted variable bias (two-variable case)

Bias in β1-hat = β2 × Cov(X1, X2) ÷ Var(X1)

True model Y = β0 + β1X1 + β2X2 + ε, with X2 omitted. The estimate converges to β1 plus this term. The sign is the sign of β2 times the sign of the correlation.

Consequences of heteroskedasticity

Coefficients: unbiased and consistent. Standard errors: invalid. OLS: not BLUE.

The standard error is not valid, so t-tests and confidence intervals are unreliable until you correct them.

Remedies

Robust (White) standard errors, or WLS/GLS

Robust standard errors keep the OLS coefficients and change only the standard errors.

Irrelevant variable

Coefficients unbiased, variances larger

Adjusted R² penalises useless regressors.

Dummy regression (intercept shift)

Y = β0 + β1 X + β2 D + ε

Base group (D = 0): intercept β0. Dummy group (D = 1): intercept β0 + β2. Slope on X is β1 for both.

Interaction model

Y = β0 + β1 X + β2 D + β3 (D × X) + ε

Slope is β1 when D = 0 and β1 + β3 when D = 1. Intercept is β0 when D = 0 and β0 + β2 when D = 1.

Number of dummies

k categories → k − 1 dummies (with an intercept)

Using k dummies plus an intercept gives perfect multicollinearity (dummy variable trap).

Test of a single coefficient

t = (β̂ − hypothesized value) ÷ SE(β̂)

Degrees of freedom are n − k − 1, where k is the number of slope coefficients.

Joint test (restricted vs unrestricted)

F = [(SSR_restricted − SSR_unrestricted) ÷ q] ÷ [SSR_unrestricted ÷ (n − k − 1)]

q is the number of restrictions, k the number of slope coefficients in the unrestricted model.

Adjusted R²

Adj R² = 1 − (1 − R²) × (n − 1) ÷ (n − k − 1)

Can fall when a variable adds little. Unlike R², it penalises extra regressors.

Information criteria

AIC = ln(SSR ÷ n) + 2k ÷ n; BIC = ln(SSR ÷ n) + k ln(n) ÷ n

Lower is better. BIC penalises extra parameters more heavily than AIC when n is 8 or more.

Quick revision

• Slope coefficient = change in Y for a one-unit change in that X, holding the other variables constant.
• OLS chooses coefficients that minimize the sum of squared residuals.
• t = (estimate − hypothesized value) ÷ standard error; degrees of freedom = n − k − 1.
• Confidence interval = estimate ± critical value × standard error.
• The F-test checks whether several coefficients are jointly zero; a single t-test cannot do this.
• Overall F = (ESS ÷ k) ÷ (SSR ÷ (n − k − 1)).
• R² never falls when you add a variable; adjusted R² can fall.
• Adjusted R² = 1 − [(n − 1) ÷ (n − k − 1)] × (1 − R²).
• Perfect multicollinearity makes OLS impossible; high multicollinearity inflates standard errors but coefficients stay unbiased.
• Classic sign of multicollinearity: high R² and significant F-test, but insignificant individual t-statistics.
• Heteroskedasticity leaves coefficients unbiased but makes the usual standard errors unreliable; robust standard errors are a common fix.
• Omitted variable bias arises when the left-out variable affects Y and is correlated with an included X.
• With a categorical variable of m categories, use m − 1 dummies and leave one as the base.

Common mistakes

• Reading a slope without saying other variables are held constant, or reading it as the simple regression effect. Fix: Always interpret each slope as a partial effect. Adding a correlated regressor can change it.
• Saying heteroskedasticity or multicollinearity makes OLS coefficients biased. Fix: Remember: both leave slopes unbiased. Heteroskedasticity distorts standard errors. Imperfect multicollinearity inflates them.
• Using n − 1 or n − 2 degrees of freedom in a multiple regression. Fix: Use n − k − 1, where k counts all slope coefficients.
• Reading the p-value as the probability that H0 is true. Fix: The p-value is the probability of a result at least this extreme if H0 were true. Reject H0 when p < α.
• Using the restricted model's degrees of freedom in the denominator. Fix: The denominator is always n − k − 1 from the unrestricted model.
• Swapping SSR_r and SSR_u, giving a negative F. Fix: Restricted SSR is never smaller than unrestricted SSR. F must be non-negative; recheck if not.
• Using k = number of coefficients including the intercept. Fix: In the FRM formula, k is the number of slope variables and the denominator is n − k − 1. Count only explanatory variables.
• Believing a higher R² means a better model after adding variables. Fix: Compare models with different numbers of regressors using adjusted R². A higher R² alone is not evidence of a better model.
• Saying imperfect multicollinearity biases the OLS coefficients. Fix: Remember OLS stays unbiased and consistent. Only the variances, and so standard errors, increase.
• Thinking multicollinearity lowers the model's R² or ruins its fit. Fix: R² is often high. The problem is separating the individual effects, not explaining the dependent variable.

Exam tips

• Questions often ask which assumption violation biases the coefficients. Only omitted correlated variables (zero conditional mean failure) do so among the usual options.
• Always compute degrees of freedom as n − k − 1 before looking up a critical value.
• In interpretation questions, watch units. A coefficient on a variable in percent versus decimals changes the answer by 100.
• Know the dummy variable trap as the standard example of perfect multicollinearity.
• Check what the null value is. A question about beta equal to 1 needs (β̂ − 1), not β̂ alone.
• Read whether the question gives a one-sided or two-sided test, and whether the table value is for one or two tails.
• Questions often give a p-value and ask for a conclusion. Compare it with α and answer in words without calculating anything.
• Use the confidence interval as a check. If the interval excludes the null value, the two-sided test must reject.