Generalised linear models: formula sheet

Key formulas

Exponential family form

f(y; θ, φ) = exp{ [yθ − b(θ)] ÷ a(φ) + c(y, φ) }

θ is the canonical parameter, φ the dispersion parameter, b(θ) the cumulant function.

Mean

E(Y) = μ = b′(θ)

Comes from differentiating the total probability once with respect to θ.

Variance

Var(Y) = a(φ) · b″(θ)

Do not forget the factor a(φ). Write b″ in terms of μ to get the variance function V(μ).

Normal(μ, σ²)

θ = μ; b(θ) = θ² ÷ 2; φ = σ²; a(φ) = φ

Mean = θ, variance = σ². V(μ) = 1.

Poisson(μ)

θ = ln μ; b(θ) = e^θ; a(φ) = 1

Mean = variance = e^θ = μ. V(μ) = μ.

Binomial count, Y ~ Bin(n, p)

θ = ln[p ÷ (1 − p)]; b(θ) = n ln(1 + e^θ); a(φ) = 1

Mean = n e^θ ÷ (1 + e^θ) = np. Variance = n e^θ ÷ (1 + e^θ)² = np(1 − p).

Gamma with mean μ and shape α

θ = −1 ÷ μ; b(θ) = −ln(−θ); φ = 1 ÷ α; a(φ) = φ

Mean = −1 ÷ θ = μ. Variance = φμ² = μ² ÷ α. V(μ) = μ².

Link and linear predictor

g(μᵢ) = ηᵢ = β₀ + β₁xᵢ₁ + … + βₚxᵢₚ, so μᵢ = g⁻¹(ηᵢ)

Linear in the parameters β. The link is monotonic and differentiable.

Exponential family form

f(y; θ, φ) = exp{ [yθ − b(θ)] ÷ a(φ) + c(y, φ) }

θ is the canonical parameter. Usually a(φ) = φ ÷ w for a prior weight w.

Mean and variance

E(Y) = μ = b′(θ); Var(Y) = a(φ) b″(θ) = a(φ) V(μ)

V(μ) is the variance function. It depends on the distribution, not on the link.

Canonical link

g(μ) = θ, so η = θ

Equivalent to g = (b′)⁻¹.

Canonical links

Normal: g(μ) = μ; Poisson: g(μ) = ln μ; Binomial: g(μ) = ln[μ ÷ (1 − μ)]; Gamma: g(μ) = 1 ÷ μ

For the binomial, μ is the probability p of success.

Variance functions

Normal: V(μ) = 1; Poisson: V(μ) = μ; Binomial: V(μ) = μ(1 − μ); Gamma: V(μ) = μ²

For the gamma, Var(Y) = φμ², where φ = 1 ÷ α with shape α.

Log link interpretation

μ = exp(β₀) × exp(β₁x₁) × …

A one-unit rise in x₁ multiplies the mean by exp(β₁).

Exponential family log-likelihood

ℓ = Σ [ (yᵢθᵢ − b(θᵢ)) ÷ a(φ) + c(yᵢ, φ) ]

With a(φ) = φ ÷ wᵢ. Mean = b′(θ), variance = a(φ) b″(θ).

Score equations

Σᵢ wᵢ (yᵢ − μᵢ) xᵢⱼ ÷ (V(μᵢ) g′(μᵢ)) = 0, for j = 1,…,p

φ cancels. Canonical link: Σᵢ wᵢ (yᵢ − μᵢ) xᵢⱼ = 0.

Working response

zᵢ = ηᵢ + (yᵢ − μᵢ) g′(μᵢ)

Computed using current estimates of ηᵢ and μᵢ.

Working weights

Wᵢ = wᵢ ÷ ( V(μᵢ) [g′(μᵢ)]² )

Recomputed at each iteration.

IRLS update

β_new = (XᵀWX)⁻¹ XᵀWz

W is the diagonal matrix of working weights.

Approximate variance of β̂

Var(β̂) ≈ φ (XᵀWX)⁻¹

W evaluated at β̂. Asymptotic result.

Pearson estimate of scale

φ̂ = (1 ÷ (n − p)) Σ wᵢ (yᵢ − μ̂ᵢ)² ÷ V(μ̂ᵢ)

p is the number of parameters in the linear predictor.

Deviance-based estimate of scale

φ̂ = D ÷ (n − p)

D is the scaled-free deviance, defined as φ times the scaled deviance. Often less reliable than Pearson's.

Scaled deviance

D* = 2[l(saturated) − l(fitted)]

Always ≥ 0. The saturated model has μ̂ᵢ = yᵢ for every observation.

Deviance and dispersion

D = φ × D*

For Poisson and binomial, φ = 1, so D = D*. For the normal, D = Σ(yᵢ − μ̂ᵢ)².

Poisson deviance

D = 2 Σ [ yᵢ ln(yᵢ ÷ μ̂ᵢ) − (yᵢ − μ̂ᵢ) ]

If yᵢ = 0 the term is 2μ̂ᵢ. If the model has an intercept with the log link, Σ(yᵢ − μ̂ᵢ) = 0, so D = 2 Σ yᵢ ln(yᵢ ÷ μ̂ᵢ).

Binomial deviance (yᵢ successes out of nᵢ)

D = 2 Σ [ yᵢ ln(yᵢ ÷ μ̂ᵢ) + (nᵢ − yᵢ) ln((nᵢ − yᵢ) ÷ (nᵢ − μ̂ᵢ)) ]

Here μ̂ᵢ = nᵢ p̂ᵢ. A term with a zero count is taken as 0 × ln 0 = 0.

Gamma deviance

D = 2 Σ [ −ln(yᵢ ÷ μ̂ᵢ) + (yᵢ − μ̂ᵢ) ÷ μ̂ᵢ ]

Variance function V(μ) = μ².

Pearson residual

rᴾᵢ = (yᵢ − μ̂ᵢ) ÷ √V(μ̂ᵢ)

Poisson: (y − μ̂) ÷ √μ̂. Divide also by √φ if you want it on the scale of the dispersion.

Pearson statistic

X² = Σ (yᵢ − μ̂ᵢ)² ÷ V(μ̂ᵢ)

Sum of squared Pearson residuals. Usually close to D when the model fits well.

Deviance residual

rᴰᵢ = sign(yᵢ − μ̂ᵢ) × √dᵢ, where D = Σ dᵢ

Sum of squared deviance residuals equals the deviance.

Goodness of fit test

D* ≈ χ² with n − p degrees of freedom

Approximate. Unreliable for small counts or ungrouped binary data.

Nested model comparison

D*(smaller) − D*(larger) ≈ χ² with q degrees of freedom

q = number of extra parameters in the larger model. If φ is unknown, use an F test with the estimated φ.

Estimated dispersion

φ̂ = X² ÷ (n − p)

Pearson-based estimate. A value well above 1 for Poisson or binomial suggests overdispersion.

Likelihood ratio statistic

LR = 2 × (ℓ₂ − ℓ₁) ~ χ²(p₂ − p₁) approximately

Model 1 (p₁ parameters) must be nested in Model 2 (p₂ parameters). ℓ is the maximised log-likelihood. Large-sample result.

Scaled deviance

D* = 2 × (ℓ_saturated − ℓ_model)

Use the scaled deviance for the chi-squared test when φ is known.

Deviance difference test (φ known)

D*₁ − D*₂ ~ χ²(p₂ − p₁)

Reject the smaller model if the difference exceeds the chi-squared critical value. For Poisson and binomial, scaled deviance equals deviance.

F-test (φ unknown)

F = [(D₁ − D₂) ÷ (p₂ − p₁)] ÷ [D₂ ÷ (n − p₂)] ~ F(p₂ − p₁, n − p₂)

D is the unscaled deviance. The denominator estimates the scale parameter. Use for normal, gamma and similar families.

Akaike Information Criterion

AIC = −2ℓ + 2p

Choose the model with the lowest AIC. p counts all estimated parameters.

Degrees of freedom of a model

Residual df = n − p

n is the number of observations (or cells), p the number of fitted parameters.

GLM structure

g(μ) = η = β0 + β1x1 + ... + βpxp

g is the link function. Y is from the exponential family with mean μ.

Poisson frequency with offset

ln(μ) = ln(e) + β0 + β1x1 + ... so μ = e × exp(β0 + β1x1 + ...)

e is exposure. The offset coefficient is fixed at 1, not estimated.

Relativity under log link

relativity = exp(β)

Multiplier on the mean for that level against the base level, other factors held fixed.

Logit link

logit(p) = ln(p ÷ (1 − p)) = η, p = exp(η) ÷ (1 + exp(η))

Gives p between 0 and 1.

Odds ratio

odds ratio = exp(β)

Multiplier on the odds p ÷ (1 − p) for a one-unit increase in x, or for a level against the base.

Pure premium

expected cost per exposure = expected frequency × expected severity

Valid when the two models are treated as independent given the rating factors.

Gamma severity

Var(Y) = φ μ²

φ is the dispersion parameter. Log link is common.

Quick revision

• A GLM has three parts: an exponential family response, a linear predictor η = Xβ, and a link function g with g(μ) = η.
• Exponential family form: f(y) = exp{[yθ − b(θ)] ÷ a(φ) + c(y, φ)}.
• Mean: E(Y) = b′(θ). Variance: Var(Y) = b″(θ) a(φ).
• The variance function V(μ) shows how variance depends on the mean, for example V(μ) = μ for Poisson.
• The canonical link sets g(μ) = θ. For Poisson it is log, for binomial it is logit, and for normal it is identity.
• The log link gives multiplicative effects: each coefficient exponentiated is a ratio change in the mean.
• Maximum likelihood estimates for GLMs usually have no closed form, so they are found by IRLS.
• Scaled deviance compares the fitted model with the saturated model, using twice the difference in log-likelihoods.
• Compare nested models by the difference in deviances. Use a chi-squared reference when the dispersion is known, and an F test when it is estimated.
• AIC = −2 × log-likelihood + 2 × (number of parameters). Lower is better, and it can compare non-nested models for the same data.
• Check residuals for patterns, since a clear pattern suggests a wrong link, a missing covariate or a wrong distribution.
• Frequency is commonly modelled with Poisson, and severity with gamma, often with a log link.

Common mistakes

• Taking θ to be the mean for every distribution. Fix: Find θ by matching the coefficient of y in the log-density. For Poisson it is ln μ. For gamma it is −1 ÷ μ.
• Forgetting a(φ) in the variance and writing Var(Y) = b″(θ). Fix: Always write Var(Y) = a(φ)·b″(θ). Check with the normal, where b″ = 1 and the variance is σ² = a(φ).
• Saying the linear predictor is linear in the covariates only. Fix: Say it is linear in the parameters β. Covariates can be transformed, such as x² or ln x.
• Modelling the mean of the transformed response, E[g(Y)], instead of g(E[Y]). Fix: A GLM links the mean: g(E[Y]) = η. The response is not transformed, and the variance follows from the chosen distribution.
• Including φ in the score equations or thinking β̂ depends on it. Fix: Note that a(φ) is a common factor in the derivative. It cancels, so β̂ is found without φ. φ matters only for standard errors and testing.
• Using n instead of n − p in the Pearson estimate of φ. Fix: Always divide by the degrees of freedom n − p, where p counts every fitted parameter including the intercept.
• Forgetting the −(y − μ̂) term in the Poisson deviance. Fix: The short form needs Σ(y − μ̂) = 0, which holds when the model has an intercept with the log link. If the model has no intercept, use the full formula.
• Using the wrong degrees of freedom for the deviance test. Fix: Absolute fit uses n − p, where p counts every fitted parameter including the intercept. Model comparison uses the difference in p between the two models.
• Using a chi-squared test on models that are not nested. Fix: Always check that one model is a special case of the other. If not, use AIC.
• Using the wrong degrees of freedom, such as the residual df of one model instead of the difference. Fix: Degrees of freedom for the test equal the number of extra parameters, which is the difference in residual df.

Exam tips

• Always show the log-density and the matching of terms. Examiners award method marks for identifying θ, b(θ), a(φ) and c(y, φ) separately.
• Memorise θ and b(θ) for the normal, Poisson, binomial and gamma. These four are the usual exam cases.
• State whether the binomial response is a count or a proportion before you write b(θ) and a(φ).
• In Paper B (computer-based), check an R or Excel result against b′(θ) and a(φ)·b″(θ) for the same parameters.
• Finish by writing the variance function V(μ). It links directly to the next topic, link functions and fitting.
• Always name all three components: random, systematic and link. Marks are usually given for each.
• If you are asked for the canonical link, derive θ from the density in the exponential family form. Showing this earns method marks.
• For a link choice, give a reason: the allowed range of μ, how effects are interpreted, or fit.