Bayesian inference: priors, posteriors, loss functions and credible intervals: formula sheet

Key formulas

Bayes' theorem for events

P(A | B) = P(B | A) × P(A) ÷ P(B)

Valid when P(B) > 0. P(B) is found by the law of total probability.

Bayes' theorem for a parameter (continuous θ)

f(θ | x) = f(x | θ) f(θ) ÷ ∫ f(x | θ) f(θ) dθ

For a discrete θ, replace the integral with a sum over θ.

Proportionality form

f(θ | x) ∝ f(θ) × L(x | θ)

Drop every factor that does not involve θ, then identify the distribution.

Likelihood for independent data

L(x | θ) = Π f(xᵢ | θ), i = 1 to n

Assumes the observations are independent given θ.

Gamma density kernel

f(θ) ∝ θ^(α−1) e^(−βθ), θ > 0

Gamma(α, β) with rate β. Mean α ÷ β.

Beta density kernel

f(θ) ∝ θ^(α−1) (1−θ)^(β−1), 0 < θ < 1

Mean α ÷ (α + β).

Bayes' theorem for densities

f(θ | x) = f(x | θ) f(θ) ÷ ∫ f(x | θ) f(θ) dθ

In practice write f(θ | x) ∝ f(x | θ) f(θ) and identify the kernel.

Beta-Binomial

Prior θ ~ Beta(α, β); data X | θ ~ Bin(n, θ); posterior θ | x ~ Beta(α + x, β + n − x)

Prior mean = α ÷ (α + β). Posterior mean = (α + x) ÷ (α + β + n).

Beta density kernel

f(θ) ∝ θ^(α−1) (1 − θ)^(β−1), 0 < θ < 1

Use this to recognise the Beta in a posterior.

Gamma-Poisson

Prior λ ~ Gamma(α, β) with density ∝ λ^(α−1) e^(−βλ); data X1..Xn | λ iid Poisson(λ); posterior λ | x ~ Gamma(α + Σxᵢ, β + n)

Here β is the rate. Prior mean = α ÷ β. Posterior mean = (α + Σxᵢ) ÷ (β + n).

Normal-Normal (known variance)

Prior μ ~ N(μ0, τ²); X1..Xn | μ iid N(μ, σ²) with σ² known; posterior μ | x ~ N(μ1, τ1²), where 1 ÷ τ1² = 1 ÷ τ² + n ÷ σ² and μ1 = τ1² (μ0 ÷ τ² + n x̄ ÷ σ²)

Precision (1 ÷ variance) adds. The posterior mean is a precision-weighted average of μ0 and x̄.

Credibility form of posterior mean

Posterior mean = Z × (data estimate) + (1 − Z) × (prior mean)

For Gamma-Poisson, Z = n ÷ (β + n) with data estimate x̄. For Normal-Normal, Z = (n ÷ σ²) ÷ (n ÷ σ² + 1 ÷ τ²).

Improper uniform prior

f(θ) ∝ 1 (non-informative)

Gives posterior ∝ likelihood. Beta(1, 1) is the proper uniform prior on (0, 1).

Posterior proportionality

f(θ | x) ∝ f(x | θ) × f(θ)

Drop every factor that does not contain θ. Then identify the distribution from what remains.

Poisson likelihood, gamma prior

X₁,…,Xₙ ~ Poisson(λ), λ ~ Gamma(α, β) ⇒ λ | x ~ Gamma(α + Σxᵢ, β + n)

Here β is the rate, so the gamma mean is α/β. Check which parametrisation the question uses.

Gamma mean, variance and mode

Mean = α/β; Variance = α/β²; Mode = (α − 1)/β for α ≥ 1

Mode formula applies when α ≥ 1. For α < 1 the density has no interior mode.

Binomial likelihood, beta prior

X ~ Bin(n, θ), θ ~ Beta(α, β) ⇒ θ | x ~ Beta(α + x, β + n − x)

Beta mean is α/(α + β). Beta mode is (α − 1)/(α + β − 2) when α, β > 1.

Normal mean, known variance, normal prior

X₁,…,Xₙ ~ N(μ, σ²), μ ~ N(μ₀, σ₀²) ⇒ μ | x ~ N(μ₁, σ₁²), where 1/σ₁² = n/σ² + 1/σ₀² and μ₁ = (n x̄/σ² + μ₀/σ₀²) × σ₁²

Precisions (1 ÷ variance) add. The posterior mean is a precision-weighted average of x̄ and μ₀.

Credibility form of posterior mean

E(θ | x) = Z × (sample estimate) + (1 − Z) × (prior mean)

Poisson-gamma: Z = n/(n + β), sample estimate x̄. Normal: Z = (n/σ²) ÷ (n/σ² + 1/σ₀²).

Bayes estimates by loss function

Quadratic loss → posterior mean; Absolute loss → posterior median; All-or-nothing loss → posterior mode

State the loss function in your answer. The estimate depends on it.

Expected posterior loss

E[L(θ̂, θ) | x] = ∫ L(θ̂, θ) f(θ | x) dθ

For a discrete θ, replace the integral with a sum over the posterior probabilities. The Bayes estimator minimises this.

Quadratic (squared error) loss

L(θ̂, θ) = (θ̂ − θ)² → θ̂ = E[θ | x]

The Bayes estimator is the posterior mean.

Absolute error loss

L(θ̂, θ) = |θ̂ − θ| → θ̂ = m, where P(θ ≤ m | x) = 0.5

The Bayes estimator is the posterior median.

All-or-nothing (0-1) loss

L(θ̂, θ) = 0 if θ̂ = θ, 1 otherwise → θ̂ = mode of f(θ | x)

For a continuous θ, the mode arises as the limit of a loss that is 0 when |θ̂ − θ| < ε and 1 otherwise, as ε → 0.

Decomposition under quadratic loss

E[(θ̂ − θ)² | x] = Var(θ | x) + (θ̂ − E[θ | x])²

Shows why the mean is optimal. The minimum expected loss is the posterior variance.

Posterior up to proportionality

f(θ | x) ∝ f(x | θ) × f(θ)

You only need the shape in θ. Drop constants that do not involve θ.

Gamma mean and mode

For Gamma(α, λ) with mean α/λ: mode = (α − 1)/λ, valid for α ≥ 1

Useful for Poisson-Gamma and Exponential-Gamma posteriors.

Posterior (proportional form)

f(θ | x) ∝ f(x | θ) × f(θ)

Identify the posterior's family by looking at the kernel, then use known quantiles of that distribution.

Credible interval definition

P(a < θ < b | x) = ∫ from a to b of f(θ | x) dθ = 1 − α

This is a probability statement about θ given the observed data.

Equal-tailed interval

P(θ < a | x) = α/2 and P(θ > b | x) = α/2

a and b are the α/2 and 1 − α/2 posterior quantiles.

HPD interval

Set {θ : f(θ | x) ≥ k}, with k chosen so that its posterior probability is 1 − α

Shortest interval of that probability. For a unimodal posterior it is a single interval with f(a | x) = f(b | x).

Normal posterior interval

μ₁ ± z(1−α/2) × σ₁, where θ | x ~ N(μ₁, σ₁²)

For 95%, z = 1.96. Equal-tailed and HPD coincide here.

Normal–normal posterior

Prior N(μ₀, σ₀²), data mean x̄ from n observations with known variance σ²: posterior precision = 1/σ₀² + n/σ²; posterior mean = (μ₀/σ₀² + n x̄/σ²) ÷ posterior precision

Precision is 1 ÷ variance. Posterior variance = 1 ÷ posterior precision.

Quick revision

• Bayes for parameters: f(θ | x) = f(x | θ) f(θ) ÷ ∫ f(x | θ) f(θ) dθ, so posterior ∝ likelihood × prior.
• In Bayesian inference, θ is random and the data x are fixed once observed.
• Only terms containing θ matter when you find the posterior kernel.
• A conjugate prior gives a posterior in the same family as the prior.
• Beta(α, β) prior with Binomial(n, θ) data and x successes gives a Beta(α + x, β + n − x) posterior.
• Gamma(α, λ) prior with Poisson data x₁,…,xₙ gives a Gamma(α + Σxᵢ, λ + n) posterior.
• Quadratic loss gives the posterior mean as the Bayes estimate.
• Absolute error loss gives the posterior median; 0-1 loss gives the posterior mode.
• The Bayes estimate minimises the expected posterior loss.
• A 95% credible interval is a range with posterior probability 0.95 that contains θ.
• A credible interval is not the same as a frequentist confidence interval, and the interpretations differ.
• A posterior mean can be written as a weighted average of the prior mean and the data estimate, with weights that shift towards the data as n grows.

Common mistakes

• Forgetting that ∝ hides a constant and treating the product as a proper density. Fix: Always identify the kernel with a named distribution, or integrate to normalise, before quoting probabilities or means.
• Keeping terms that do not involve θ inside the kernel. Fix: Remove everything free of θ. Then the powers of θ and the exponent are easy to read.
• Using the Gamma scale parameter as the rate (or the other way round) in Gamma-Poisson. Fix: Write the density first. If it is ∝ λ^(α−1) e^(−βλ), β is a rate and you add n to it. If it is e^(−λ/θ), the scale is θ and 1 ÷ θ becomes 1 ÷ θ + n.
• Adding the number of trials to β in Beta-Binomial instead of the number of failures. Fix: Derive the exponents: θ^(α+x−1) (1−θ)^(β+n−x−1). So β becomes β + n − x.
• Using x̄ instead of Σx when updating a gamma prior. Fix: Write Gamma(α + Σx, β + n) every time. Sanity-check that posterior mean (α + Σx)/(β + n) lies between x̄ and α/β.
• Mixing up rate and scale for the gamma distribution. Fix: Read the question's density or the Tables. Write the mean of the prior to confirm which form is used before updating.
• Giving the posterior mean when the question asks for absolute loss or all-or-nothing loss. Fix: Underline the loss function first. Write the pairing mean, median, mode next to the working.
• Using the prior mean or the maximum likelihood estimate instead of the posterior summary. Fix: Always derive the posterior first. A Bayes estimator is a function of the posterior.
• Interpreting a 95% confidence interval as a 95% probability that θ is in the interval. Fix: Say the 95% describes the long-run coverage of the method. Reserve the probability statement about θ for credible intervals.
• Using the equal-tailed interval when the question asks for HPD, or assuming they are always equal. Fix: Check the shape of the posterior. If it is skewed, the HPD interval is shorter and has equal density at both ends, not equal tail probabilities.

Exam tips

• In written questions, show prior, likelihood, product and kernel as separate lines. Marks are given for each stage.
• Always state the final distribution with its name and parameters. Do not stop at 'proportional to'.
• For discrete θ, build a small table of prior, likelihood, product and posterior. It reduces arithmetic slips.
• In MCQs, check whether the question wants the posterior mean, the posterior distribution or a probability. These give different numbers.
• Compare Bayesian and classical ideas in words when asked. Mention the prior, θ as random, and the meaning of the interval.
• In written questions, always show the line posterior ∝ likelihood × prior and name the kernel. Method marks are given for this step even if the arithmetic goes wrong.
• Check the parameterisation of the Gamma, and of the Exponential, before you update. The IAI formulae book and the question wording set which one applies.
• Expect MCQs asking you to identify the posterior or its mean from a short data summary. Use the add-the-data shortcut and do the check that the posterior mean lies between the prior mean and the data estimate.