Estimators and their properties: formula sheet

Key formulas

Estimator for a parameter

θ̂ = g(X₁, X₂, ..., Xₙ)

A function of the random sample. It is a random variable. Uppercase X is used.

Estimate

θ̂ = g(x₁, x₂, ..., xₙ)

The same function applied to observed values. It is a single number. Lowercase x is used.

Sample mean

X̄ = (1 ÷ n) Σ Xᵢ

Common estimator of the population mean μ.

Sample variance

S² = (1 ÷ (n − 1)) Σ (Xᵢ − X̄)²

Common estimator of σ². Uses divisor n − 1.

Mean and variance of X̄

E(X̄) = μ and Var(X̄) = σ² ÷ n

For independent observations with common mean μ and variance σ².

Standard error

SE(θ̂) = √Var(θ̂)

The standard deviation of the estimator's sampling distribution.

Bias

bias(θ̂) = E(θ̂) − θ

Defined from the sampling distribution. Studied further under unbiasedness.

Bias

bias(θ̂) = E(θ̂) − θ

The expectation is taken over the sampling distribution of θ̂. θ is treated as fixed.

Unbiased estimator

E(θ̂) = θ for all θ in the parameter space

It must hold for every possible value of θ, not just one.

Asymptotically unbiased

E(θ̂ₙ) → θ as n → ∞

Equivalently, the bias tends to 0 as n → ∞.

Expectation of sample mean

E(X̄) = μ, Var(X̄) = σ² ÷ n

For i.i.d. observations with mean μ and variance σ². The variance result also needs independence.

Sample variance

S² = Σ(Xᵢ − X̄)² ÷ (n − 1), E(S²) = σ²

Uses i.i.d. observations with finite variance.

Divisor n version

E[Σ(Xᵢ − X̄)² ÷ n] = (n − 1)σ² ÷ n

Bias is −σ² ÷ n, which tends to 0, so it is asymptotically unbiased.

Key identity

Σ(Xᵢ − X̄)² = ΣXᵢ² − nX̄²

Used in the proof that S² is unbiased.

Variance and second moment

E(Y²) = Var(Y) + [E(Y)]²

Used to find E(Xᵢ²) and E(X̄²).

Bias

bias(θ̂) = E[θ̂] − θ

Zero for an unbiased estimator. It can be positive or negative. Always subtract θ from the expected value.

Mean squared error

MSE(θ̂) = E[(θ̂ − θ)²]

The definition. It is the average squared error around the true value θ.

MSE decomposition

MSE(θ̂) = Var(θ̂) + [bias(θ̂)]²

The main working formula. Prove it by writing θ̂ − θ = (θ̂ − E[θ̂]) + (E[θ̂] − θ) and expanding. The cross term has zero expectation.

Unbiased case

If E[θ̂] = θ, then MSE(θ̂) = Var(θ̂)

Compare unbiased estimators by variance.

Relative efficiency

eff(θ̂1 relative to θ̂2) = MSE(θ̂2) ÷ MSE(θ̂1)

Above 1 means θ̂1 is better. For two unbiased estimators this is Var(θ̂2) ÷ Var(θ̂1).

Variance of sample mean

Var(X̄) = σ² ÷ n

For independent observations with common variance σ². X̄ is unbiased for μ.

Variance estimators for a normal sample

S² (divisor n−1): bias 0, MSE = 2σ⁴ ÷ (n−1). Divisor n: bias = −σ² ÷ n, MSE = (2n−1)σ⁴ ÷ n²

Valid for a random sample from a normal distribution. The divisor n−1 is unbiased but the divisor n has smaller MSE.

MSE and consistency

MSE(θ̂) → 0 as n → ∞ implies θ̂ is consistent

This is a sufficient condition. It is not necessary.

Definition of consistency

θ̂ₙ is consistent for θ if P(|θ̂ₙ − θ| > ε) → 0 as n → ∞, for every ε > 0

This is convergence in probability, written θ̂ₙ →p θ.

Mean squared error

MSE(θ̂) = E[(θ̂ − θ)²] = Var(θ̂) + [bias(θ̂)]²

Where bias(θ̂) = E(θ̂) − θ.

Sufficient condition for consistency

bias(θ̂ₙ) → 0 and Var(θ̂ₙ) → 0 as n → ∞ ⇒ θ̂ₙ is consistent

Equivalent to MSE(θ̂ₙ) → 0. Sufficient, not necessary.

Markov/Chebyshev link

P(|θ̂ₙ − θ| > ε) ≤ MSE(θ̂ₙ) ÷ ε²

Markov's inequality applied to (θ̂ₙ − θ)², a form of Chebyshev's inequality. This is why MSE → 0 forces convergence in probability.

Variance of the sample mean

Var(X̄) = σ² ÷ n

For independent observations with common variance σ². Used to show X̄ is consistent for μ.

Factorisation criterion

f(x | θ) = g(t(x), θ) × h(x)

T = t(X) is sufficient for θ if the likelihood factorises this way, with h(x) free of θ. Include any indicator for a θ-dependent support in g.

Log-likelihood score

U(θ) = ∂/∂θ log L(θ)

For a regular model, E[U(θ)] = 0.

Fisher information (two forms)

I_n(θ) = E[(∂/∂θ log L)²] = −E[∂²/∂θ² log L]

Both forms are equal under regularity conditions. The second form is usually quicker. Take the expectation after differentiating twice.

Information from an iid sample

I_n(θ) = n × I_1(θ)

I_1 comes from a single observation. Do not forget the factor n.

CRLB for θ

Var(θ̂) ≥ 1 ÷ I_n(θ)

Applies to unbiased estimators of θ under regularity conditions.

CRLB for a function g(θ)

Var(T) ≥ [g′(θ)]² ÷ I_n(θ)

Applies when T is unbiased for g(θ).

Efficiency

e(T) = CRLB ÷ Var(T)

Between 0 and 1 for an unbiased estimator in a regular model. It equals 1 when the bound is attained.

Quick revision

• An estimator is a random variable; an estimate is the number from one observed sample.
• Bias of θ̂ = E(θ̂) − θ. The estimator is unbiased if this is 0 for every value of θ.
• MSE(θ̂) = E[(θ̂ − θ)²] = Var(θ̂) + [bias(θ̂)]².
• For unbiased estimators, MSE equals the variance.
• A lower MSE means a better estimator; a biased estimator can beat an unbiased one on MSE.
• Sample variance with divisor n − 1 is unbiased for σ²; the divisor n version is biased.
• Consistency: the estimator converges to the true value as n → ∞. Bias → 0 and variance → 0 together are enough.
• A statistic is sufficient if it carries all the information in the sample about the parameter.
• Factorisation criterion: the likelihood splits as g(t, θ) × h(x), where t is the statistic.
• CRLB: for an unbiased estimator, Var(θ̂) ≥ 1 ÷ I(θ), where I(θ) is the Fisher information of the whole sample, under the usual regularity conditions.
• An unbiased estimator that reaches the CRLB is efficient, but many good estimators do not reach it.
• State the conditions behind any rule you quote, such as unbiasedness for the CRLB.

Common mistakes

• Using the words estimator and estimate as if they mean the same thing. Fix: Estimator is the random rule with uppercase X. Estimate is the number from data with lowercase x. Write which one you mean.
• Treating the parameter as random when finding the sampling distribution. Fix: In classical estimation the parameter is fixed. The randomness comes only from the sample.
• Writing E(X̄²) = μ² when finding E(S²). Fix: Use E(X̄²) = Var(X̄) + μ² = σ²÷n + μ². The extra σ²÷n term is what produces the n − 1.
• Saying an unbiased estimator is always close to the true value. Fix: Unbiased means correct on average only. A large variance can still give poor single estimates. Mention variance or mean squared error when comparing estimators.
• Writing MSE = variance + bias instead of bias squared. Fix: Always write MSE = Var + (bias)². Check units: variance is in squared units, so bias must be squared too.
• Comparing biased estimators by variance alone. Fix: Compute full MSE for every estimator unless both are unbiased.
• Saying an unbiased estimator is automatically consistent. Fix: Unbiased concerns the mean at fixed n. Always check that the variance also tends to zero.
• Saying a biased estimator cannot be consistent. Fix: Only bias that does not vanish matters. If bias → 0 as n → ∞ and variance → 0, the estimator is consistent.
• Using I_1(θ) instead of I_n(θ) in the bound, so the answer has no n. Fix: After finding I_1, write I_n = n × I_1 on a new line. Check that the final bound shrinks as n grows.
• Applying the CRLB to Uniform(0, θ) or another model whose range depends on θ. Fix: Always check the range of x first. If it depends on θ, say the CRLB does not apply and move on.

Exam tips

• Match notation to meaning. Uppercase X for estimators, lowercase x for estimates. Examiners award marks for this in written answers.
• When asked for a sampling distribution, state the distribution name and both parameters, not only the mean.
• Always state your assumptions, such as independence and identical distribution, before using Var(X̄) = σ²/n.
• In MCQs, check whether a quantity includes an unknown parameter. If it does, it is not a statistic.
• In the computer-based paper, compute the estimate from the data with the stated formula and show the formula in your working before the number.
• Show the identity Σ(Xᵢ − X̄)² = ΣXᵢ² − nX̄² and each expectation line by line. Marks are given for the method.
• State assumptions at the start, for example independent and identically distributed with finite variance.
• In MCQs on linear estimators, check whether the weights sum to 1 before anything else.