Random sampling and sampling distributions: formula sheet

Key formulas

iid random sample

X₁, …, Xₙ iid with the distribution of X

Independent and identically distributed. Each Xᵢ has the same mean μ and variance σ² as the population.

Joint density of an iid sample

f(x₁, …, xₙ) = f(x₁) × f(x₂) × … × f(xₙ)

Uses independence. It is the starting point for the likelihood function.

Sample mean

X̄ = (1/n) Σ Xᵢ

A statistic. Its observed value is x̄.

Sample variance

S² = Σ (Xᵢ − X̄)² ÷ (n − 1)

A statistic with divisor n − 1. It is an unbiased estimator of σ² for an iid sample.

Mean and variance of X̄ for an iid sample

E(X̄) = μ and Var(X̄) = σ² ÷ n

Needs independence and finite variance. Standard error of X̄ is σ ÷ √n.

Number of samples of size n

With replacement: Nⁿ ordered samples. Without replacement: N!/(n!(N − n)!) unordered samples

N is the population size. Each is equally likely under simple random sampling.

Sample mean

X̄ = (1/n) Σ Xᵢ

Sum over i = 1 to n.

Sample variance

S² = Σ(Xᵢ − X̄)² ÷ (n − 1) = (ΣXᵢ² − nX̄²) ÷ (n − 1)

The second form is faster for calculation. Know that the divisor is n − 1.

Expectation of X̄

E[X̄] = μ

Needs only that each Xᵢ has mean μ.

Variance of X̄

Var(X̄) = σ² ÷ n

Needs independence (or at least uncorrelated) and common variance σ².

Unbiasedness of S²

E[S²] = σ²

Holds for iid samples with finite variance, not only normal ones.

Key identity

Σ(Xᵢ − μ)² = Σ(Xᵢ − X̄)² + n(X̄ − μ)²

Used in the proof that E[S²] = σ².

Normal population result

(n − 1)S² ÷ σ² ~ χ² with n − 1 degrees of freedom

Normal population only. Then Var(S²) = 2σ⁴ ÷ (n − 1).

Distribution of X̄ (normal population)

X̄ ~ N(μ, σ²/n)

Exact for normal data; approximate for large n by the Central Limit Theorem.

Mean and variance of the sample mean

E(X̄) = μ ; Var(X̄) = σ² ÷ n

Needs iid observations with finite variance. Holds for any distribution.

Standard error

SE(X̄) = σ ÷ √n

Use s in place of σ only when σ is unknown, and then the t distribution is relevant for normal data.

Mean and variance of a sum

E(Sₙ) = nμ ; Var(Sₙ) = nσ²

Sₙ = X₁ + ... + Xₙ. Variance adds only for independent variables.

Normal population

X̄ ~ N(μ, σ² ÷ n)

Exact for any n when each Xᵢ ~ N(μ, σ²).

Central Limit Theorem

(X̄ − μ) ÷ (σ ÷ √n) → N(0, 1) as n → ∞

Requires iid with finite variance. It is an approximation for finite n.

CLT for a sum

Z = (Sₙ − nμ) ÷ (σ √n) ≈ N(0, 1)

Use for totals such as aggregate claims.

Continuity correction

P(Sₙ ≤ k) ≈ Φ((k + 0.5 − nμ) ÷ (σ√n))

Use only when the sum is integer valued, for example binomial or Poisson.

Chi-square definition

χ²(k) = Z₁² + Z₂² + … + Z_k², Zᵢ independent N(0,1)

Mean k, variance 2k. It is Gamma(k/2, 1/2).

Sample variance

S² = Σ(Xᵢ - X̄)² ÷ (n - 1)

Divisor n-1. Check which divisor a question uses.

Distribution of scaled sample variance

(n - 1)S²/σ² ~ χ²(n - 1)

Needs a random sample from a normal population. Then E[S²] = σ² and Var(S²) = 2σ⁴/(n-1).

Independence

X̄ and S² are independent

True for normal samples. Not true in general.

Chi-square MGF

M(t) = (1 - 2t)^(-k/2), for t < 1/2

Use it to prove sums of independent chi-squares are chi-square with added degrees of freedom.

t distribution definition

T = Z ÷ √(W/ν) ~ t(ν), Z ~ N(0,1), W ~ χ²(ν) independent

Mean 0 for ν > 1. Variance ν/(ν-2) for ν > 2.

t statistic for the mean

(X̄ - μ) ÷ (S/√n) ~ t(n - 1)

Use when σ is unknown and the population is normal.

F distribution definition

F = (U/m) ÷ (V/n) ~ F(m, n), U ~ χ²(m), V ~ χ²(n) independent

Order of degrees of freedom matters: numerator first.

Ratio of sample variances

(S₁²/σ₁²) ÷ (S₂²/σ₂²) ~ F(n₁ - 1, n₂ - 1)

Two independent normal samples.

Useful links

1/F(m, n) ~ F(n, m); T² ~ F(1, ν) if T ~ t(ν); F_α(m, n) = 1 ÷ F_(1-α)(n, m)

Here F_α denotes the upper α point. Use the reciprocal rule to read lower points from tables.

Cdf of the maximum

F(n)(x) = P(X(n) ≤ x) = [F(x)]ⁿ

Needs iid observations. Pdf: n [F(x)]ⁿ⁻¹ f(x).

Cdf of the minimum

F(1)(x) = 1 − [1 − F(x)]ⁿ

Pdf: n [1 − F(x)]ⁿ⁻¹ f(x).

Cdf of the kth order statistic

F(k)(x) = Σ (j = k to n) C(n, j) [F(x)]ʲ [1 − F(x)]ⁿ⁻ʲ

At least k of the n values are at most x.

Pdf of the kth order statistic

f(k)(x) = n! ÷ [(k − 1)! (n − k)!] × [F(x)]ᵏ⁻¹ [1 − F(x)]ⁿ⁻ᵏ f(x)

For a continuous distribution. Valid for k = 1, ..., n.

Minimum of exponentials

If Xᵢ ~ Exp(λ) iid, then X(1) ~ Exp(nλ)

Mean of the minimum is 1 ÷ (nλ).

Uniform order statistics

If Xᵢ ~ U(0,1) iid, X(k) ~ Beta(k, n − k + 1), E[X(k)] = k ÷ (n + 1)

Variance = k(n − k + 1) ÷ [(n + 1)²(n + 2)].

Sample median

n odd: X((n+1)/2). n even: [X(n/2) + X(n/2 + 1)] ÷ 2

Definitions of sample quantiles differ for other p. Use the one given in the question.

Quick revision

• A statistic is a function of the sample, so it is a random variable with its own distribution.
• For an i.i.d. sample with mean μ and variance σ²: E(X̄) = μ and Var(X̄) = σ² ÷ n.
• The sample variance S² = Σ(Xᵢ − X̄)² ÷ (n − 1) is an unbiased estimator of σ².
• For a normal population, X̄ ~ N(μ, σ² ÷ n) exactly, for any n.
• For a normal population, (n − 1)S² ÷ σ² ~ χ² with n − 1 degrees of freedom, and X̄ and S² are independent.
• The Central Limit Theorem: for i.i.d. variables with finite variance, (X̄ − μ) ÷ (σ ÷ √n) is approximately N(0, 1) for large n.
• If Z ~ N(0, 1) and V ~ χ²(k) are independent, then Z ÷ √(V ÷ k) ~ t with k degrees of freedom.
• For a normal sample with unknown σ, (X̄ − μ) ÷ (S ÷ √n) ~ t with n − 1 degrees of freedom.
• If U ~ χ²(m) and V ~ χ²(k) are independent, then (U ÷ m) ÷ (V ÷ k) ~ F with (m, k) degrees of freedom.
• For i.i.d. X with distribution function F: P(max ≤ x) = [F(x)]ⁿ and P(min > x) = [1 − F(x)]ⁿ.
• Sample quantiles come from the ordered sample. Check the interpolation rule your question or R uses.

Common mistakes

• Calling X̄ a parameter or μ a statistic. Fix: μ belongs to the population and is fixed. X̄ comes from the sample and varies. Ask which one changes if you redraw the sample.
• Treating draws without replacement as independent. Fix: Without replacement, the draws are dependent. They are identically distributed but not independent. The iid model is only an approximation when N is much larger than n.
• Dividing by n instead of n − 1 when finding S². Fix: For a sample used to estimate σ², divide by n − 1. Use n only if the question defines the variance of the data as a population or asks for the maximum likelihood estimate under a normal model.
• Writing Var(X̄) = σ² instead of σ²/n. Fix: Var(ΣXᵢ) = nσ², then divide by n² to get σ²/n.
• Using σ as the standard deviation of X̄ instead of σ/√n. Fix: Underline the word mean or sum in the question. For X̄, always divide σ by √n.
• Writing Var(Sₙ) = n²σ² or Var(X̄) = σ². Fix: Sum of n independent variables: variance nσ². Then X̄ = Sₙ/n, so the variance is nσ²/n² = σ²/n.
• Using n instead of n-1 as the degrees of freedom for (n-1)S²/σ² or for the t statistic. Fix: Write the pivot with its degrees of freedom before reading any table: χ²(n-1) or t(n-1).
• Using the normal table when σ is unknown and n is small. Fix: If S replaces σ, use t(n-1). The t has heavier tails, so its critical values are larger than the normal ones.
• Writing the cdf of the minimum as [F(x)]ⁿ, or the maximum as 1 − [1 − F(x)]ⁿ. Fix: Ask: max ≤ x needs all values ≤ x, so use F(x)ⁿ. Min > x needs all values > x, so use survival to the power n.
• Forgetting the factor n in the pdf of the minimum or maximum. Fix: Differentiate [F(x)]ⁿ properly to get n [F(x)]ⁿ⁻¹ f(x). Check that the pdf integrates to 1.

Exam tips

• In MCQs, classify each quantity as parameter or statistic first. Check whether any unknown parameter appears in it.
• Always state the iid assumption in written answers. Marks are often given for stating it.
• Keep capital and small letters correct. Examiners notice X̄ versus x̄.
• For finite populations, decide ordered or unordered and with or without replacement before counting.
• In Paper B, a sample in R is a vector of draws. Say clearly what the population distribution is and what statistic you compute.
• Examiners often ask for the proof of E[S²] = σ². Learn the ΣXᵢ² − nX̄² route; it is the shortest.
• Always state iid and finite variance before using σ²/n.
• In multiple-choice questions check whether the data are described as a sample or a whole population before choosing the divisor.