Least Squares Estimation of Slope and Intercept

Understand Least Squares Estimation of Parameters

Key rules to remember

Sxx

Sxx = Σ(xi − x̄)² = Σxi² − n x̄²

Use the second form for calculations. It needs only Σx and Σx².

Sxy

Sxy = Σ(xi − x̄)(yi − ȳ) = Σxiyi − n x̄ ȳ

Sign of Sxy is the sign of the slope.

Syy

Syy = Σ(yi − ȳ)² = Σyi² − n ȳ²

Needed for σ̂², SSE and R².

Normal equations

Σy = nα + βΣx; Σxy = αΣx + βΣx²

Obtained by setting ∂S/∂α = 0 and ∂S/∂β = 0.

Least squares slope

β̂ = Sxy ÷ Sxx

Also equal to Σ(xi − x̄)yi ÷ Sxx.

Least squares intercept

α̂ = ȳ − β̂ x̄

Fitted line passes through (x̄, ȳ).

Mean of estimators

E(β̂) = β and E(α̂) = α

Both are unbiased under the model assumptions.

Variance of slope

Var(β̂) = σ² ÷ Sxx

Larger spread in x gives smaller variance.

Variance of intercept

Var(α̂) = σ² (1/n + x̄²/Sxx)

Smallest when x̄ = 0.

Covariance

Cov(α̂, β̂) = −σ² x̄ ÷ Sxx

Zero only if x̄ = 0.

Residual sum of squares

SSE = Syy − Sxy² ÷ Sxx

Equals Σ(yi − ŷi)².

Unbiased estimator of σ²

σ̂² = SSE ÷ (n − 2)

Divide by n − 2 because two parameters are estimated.

How to solve Least Squares Estimation of Parameters questions

1. 1Write down the model and assumptions: Yi = α + βxi + εi, with E(εi) = 0, Var(εi) = σ², uncorrelated errors.
2. 2List n, Σx, Σy, Σx², Σxy and, if needed, Σy². Compute x̄ and ȳ.
3. 3Compute Sxx = Σx² − n x̄² and Sxy = Σxy − n x̄ ȳ. Compute Syy if the question needs SSE or σ̂².
4. 4Calculate β̂ = Sxy ÷ Sxx, then α̂ = ȳ − β̂ x̄. State the fitted line ŷ = α̂ + β̂x.
5. 5If asked for σ̂², compute SSE = Syy − Sxy² ÷ Sxx, then divide by n − 2.
6. 6For standard errors, substitute σ̂² into Var(β̂) = σ²/Sxx or Var(α̂) = σ²(1/n + x̄²/Sxx), then take the square root.
7. 7For a derivation or proof, write β̂ as a linear combination Σ ci Yi with ci = (xi − x̄)/Sxx, then use Σci = 0 and Σci xi = 1.
8. 8Check: sense of the sign of the slope, and that the line passes through (x̄, ȳ). Give the answer with units and an interpretation.

Quickest way: Summation-table shortcut

Common mistakes in Least Squares Estimation of Parameters

• Dividing by n − 1 or n instead of n − 2 when estimating σ².

  Students copy the sample variance rule, which loses only one degree of freedom.

  Fix: Regression estimates two parameters, α and β. So σ̂² = SSE ÷ (n − 2) is the unbiased estimator.

• Using Σx² in place of Sxx, or Σxy in place of Sxy.

  The raw sums look similar to the corrected sums and the subtraction of n x̄² or n x̄ ȳ is forgotten.

  Fix: Always write Sxx = Σx² − n x̄² and Sxy = Σxy − n x̄ ȳ before computing the slope.

• Mixing up the roles of x and y, giving Sxy ÷ Syy as the slope.

  Students forget which variable is the response. Regressing x on y gives a different line.

  Fix: The slope of y on x is Sxy ÷ Sxx. The denominator is always the sum of squares of the explanatory variable.

• Writing Var(α̂) = σ²/n or Var(α̂) = σ²/Sxx.

  Students half-remember the variance of ȳ and of β̂.

  Fix: Remember Var(α̂) = σ²(1/n + x̄²/Sxx). It is the variance of ȳ plus the extra uncertainty from β̂.

• Saying the estimators are unbiased only if the errors are normal.

  Confusing the unbiasedness result with the distribution result.

  Fix: Unbiasedness, the variance formulas and Gauss-Markov need only zero mean errors, constant variance and uncorrelated errors. Normality is needed for exact distributions, t tests and the MLE link.

• Treating the estimate as exact when predicting far outside the data.

  The formula gives a number for any x.

  Fix: State that extrapolation beyond the observed range of x relies on the linear model still holding, which the data cannot confirm.

Worked examples