Sufficiency and Cramér-Rao Lower Bound Explained

Understand Sufficiency and Cramér-Rao Lower Bound

Key rules to remember

Factorisation criterion

f(x | θ) = g(t(x), θ) × h(x)

T = t(X) is sufficient for θ if the likelihood factorises this way, with h(x) free of θ. Include any indicator for a θ-dependent support in g.

Log-likelihood score

U(θ) = ∂/∂θ log L(θ)

For a regular model, E[U(θ)] = 0.

Fisher information (two forms)

I_n(θ) = E[(∂/∂θ log L)²] = −E[∂²/∂θ² log L]

Both forms are equal under regularity conditions. The second form is usually quicker. Take the expectation after differentiating twice.

Information from an iid sample

I_n(θ) = n × I_1(θ)

I_1 comes from a single observation. Do not forget the factor n.

CRLB for θ

Var(θ̂) ≥ 1 ÷ I_n(θ)

Applies to unbiased estimators of θ under regularity conditions.

CRLB for a function g(θ)

Var(T) ≥ [g′(θ)]² ÷ I_n(θ)

Applies when T is unbiased for g(θ).

Efficiency

e(T) = CRLB ÷ Var(T)

Between 0 and 1 for an unbiased estimator in a regular model. It equals 1 when the bound is attained.

How to solve Sufficiency and Cramér-Rao Lower Bound questions

1. 1Write the density or probability function for one observation, then the likelihood for the whole sample as a product.
2. 2For sufficiency, group all terms with θ and split off the terms with no θ. Identify t(x) and name g and h. Check the range of x for θ.
3. 3State the sufficient statistic clearly, for example Σxᵢ or max xᵢ. Any one-to-one function of it is also sufficient.
4. 4For the CRLB, check the regularity condition: the range of x must not depend on θ.
5. 5Take log L and differentiate twice with respect to θ.
6. 6Take the negative expectation of the second derivative to get I_n(θ). Use E[X] or E[ΣX] with the true parameter. Check that I_n = n × I_1.
7. 7Compute the bound as 1 ÷ I_n(θ), or [g′(θ)]² ÷ I_n(θ) if the target is g(θ).
8. 8Compare with Var of the estimator. Show it is unbiased first. If the variances are equal, state that it attains the bound and is the MVUE. Efficiency = bound ÷ variance.

Quickest way: Fast CRLB for one-parameter families

Common mistakes in Sufficiency and Cramér-Rao Lower Bound

• Using I_1(θ) instead of I_n(θ) in the bound, so the answer has no n.

  You compute information from a single observation and then stop.

  Fix: After finding I_1, write I_n = n × I_1 on a new line. Check that the final bound shrinks as n grows.

• Applying the CRLB to Uniform(0, θ) or another model whose range depends on θ.

  The calculation looks routine, so the regularity condition is not checked.

  Fix: Always check the range of x first. If it depends on θ, say the CRLB does not apply and move on.

• Using the plain bound 1 ÷ I_n for an estimator of g(θ) such as 1/θ or σ².

  You remember the formula for θ and forget the numerator.

  Fix: Identify the parameter being estimated. Use [g′(θ)]² ÷ I_n(θ) whenever the target is not θ itself.

• Leaving θ in h(x) during factorisation, or dropping the indicator function.

  You treat the support as unimportant.

  Fix: Put every θ term, including indicators such as I(max x ≤ θ), into g. h must be free of θ.

• Forgetting the minus sign or expectation in −E[∂²/∂θ² log L], giving a negative or random information.

  You stop after the second derivative.

  Fix: Information is always positive. Differentiate twice, change sign, then take the expectation.

• Treating an unbiased estimator that does not reach the CRLB as 'not the MVUE'.

  You assume the bound is always achievable.

  Fix: Say only that it is not efficient in the CRLB sense. Being above the bound does not by itself rule out being the MVUE.

Worked examples