Linear Regression Models for IAI CS1 Actuarial Statistics

What this chapter covers

Linear regression models: topics in the order to study them

1. 1Simple Linear Regression Model and AssumptionsEverything later depends on the model form and the assumptions on the errors, so learn these first.
2. 2Least Squares Estimation of ParametersYou need the estimators, their means and their variances before you can test or interval anything.
3. 3Inference: Confidence Intervals and Hypothesis TestsOnce you know the distribution of the estimators, you can build t-based intervals and tests, including prediction.
4. 4Goodness of Fit and ANOVA for RegressionThis splits the total sum of squares and links the F test to the t test, which you can only follow after inference.
5. 5Multiple Linear Regression and Matrix FormIt generalises the simple case, so the simple results act as a check on the matrix results.
6. 6Residuals and Model DiagnosticsStudy it last so you can judge the assumptions in all the earlier models and tie the chapter together.

How to prepare Linear regression models

1. Write the model and the error assumptions from memory: errors independent, mean zero, constant variance σ², and normal for inference. Say which results need normality and which do not.
2. Derive the least squares estimators for the simple model. Learn Sxx, Sxy and Syy, and the forms β̂1 = Sxy ÷ Sxx and β̂0 = ȳ − β̂1x̄. Do this until it takes a few minutes.
3. Practise inference with real numbers. Use the estimate of σ² with n − 2 degrees of freedom, build the t statistic, and give confidence intervals for the slope, the mean response and a new observation. Be clear why the last one is wider.
4. Build the ANOVA table by hand: SST, SSR, SSE, degrees of freedom, mean squares and F. Check that R² = SSR ÷ SST and see how F relates to the square of the slope t statistic in the simple model.
5. Learn the matrix form: β̂ = (XᵀX)⁻¹Xᵀy. Practise setting up the design matrix, then interpret coefficients, adjusted R² and the F test for the whole model. Use n − p degrees of freedom, where p counts all parameters including the intercept.
6. Fit models in R with lm() and read summary() and anova() output. Practise plotting residuals against fitted values and a normal Q-Q plot, then write two or three sentences of comment on what you see and what you would change.
7. Finish with timed past-paper questions. Check each answer for correct degrees of freedom, units and a stated conclusion.

Common mistakes in Linear regression models

• Using the wrong degrees of freedom for t tests and σ̂².

  Fix: Count the estimated parameters p, including the intercept, and use n − p. Write this down before you look up any table.

• Mixing up the confidence interval for the mean response and the prediction interval for a new observation.

  Fix: Ask what is being estimated. For a single new value, add σ² to the variance, which gives the extra 1 inside the square root. Check that your prediction interval is the wider one.

• Treating a large R² as proof that the model is correct.

  Fix: Say that R² measures the share of variation explained, nothing more. Always check the residual plots for curvature, outliers and changing spread.

• Stating a conclusion from a test without linking it to the hypotheses and the context.

  Fix: State H0 and H1, the statistic, the critical value or p-value, the decision and a one-line conclusion in context.

• Confusing the t test on one coefficient with the F test for the whole multiple regression model.

  Fix: Remember that the F test asks whether any explanatory variable helps, while each t test asks about one coefficient given the others in the model.

• Reading R output without stating assumptions or commenting on diagnostics in Paper B.

  Fix: For each output, write what the number means, whether the assumptions look reasonable from the plots, and what you would do next.

Last-day revision: Linear regression models

• ✓Model: Yi = β0 + β1xi + εi, with errors independent, mean 0 and constant variance σ².
• ✓For inference, assume errors are normal: εi ~ N(0, σ²).
• ✓β̂1 = Sxy ÷ Sxx and β̂0 = ȳ − β̂1x̄, where Sxy = Σ(xi − x̄)(yi − ȳ) and Sxx = Σ(xi − x̄)².
• ✓Var(β̂1) = σ² ÷ Sxx. Replace σ² with its estimate to get the standard error.
• ✓σ̂² = SSE ÷ (n − 2) in simple regression, and SSE ÷ (n − p) with p parameters in general.
• ✓The t statistic for a slope is (β̂1 − β1) ÷ se(β̂1), with n − 2 degrees of freedom in simple regression.
• ✓A prediction interval for a new observation is wider than the confidence interval for the mean response, because it adds the error variance.
• ✓SST = SSR + SSE, and R² = SSR ÷ SST.
• ✓In simple regression, the F statistic equals the square of the slope t statistic.
• ✓Multiple regression: β̂ = (XᵀX)⁻¹Xᵀy, and Var(β̂) = σ²(XᵀX)⁻¹.
• ✓Adjusted R² penalises extra explanatory variables, while plain R² never falls when you add one.
• ✓Residuals against fitted values should show no pattern. A curve suggests a missing term, and a funnel suggests non-constant variance.

Linear regression models practice questions

• A regression through five points gives residual sum of squares 18 for the model y = α + βx + ε with errors N(0, σ²). What is the unbiased es…
• In a multiple linear regression with p parameters (including the intercept) fitted to n observations, H is the hat matrix X(XᵀX)⁻¹Xᵀ. Which …
• Data (x, y): (1, 3), (2, 5), (3, 7), (4, 9) are fitted by least squares as y = a + bx. What is the residual sum of squares?
• In a simple linear regression fitted to 12 observations, the total sum of squares is 480 and the regression (explained) sum of squares is 36…
• In a multiple regression with p = 4 parameters (including intercept) and n = 20 observations, one observation has standardised residual r = …
• In the multiple linear regression model Y = Xβ + ε, where X is an n × p design matrix of full column rank (including a column of ones) and t…
• A regression with an intercept and 3 explanatory variables is fitted to n = 24 observations. The total sum of squares is 480 and the residua…
• In a simple linear regression with n = 27 observations, the total sum of squares is 500 and the regression sum of squares is 300. What is th…

Linear regression models in other exams

The same ground in other exams, if you are preparing for more than one or want another angle on it.

• CFA Level IApplications of Simple Linear Regression in Finance
• FRM Part ILinear Regression
• CFA Level IIBasics of Multiple Regression and Underlying Assumptions
• CMA FinalBusiness Forecasting Models - Time Series and Regression Analysis