CFA Level II Exam · Big Data Projects

Model Fit and Tuning in Big Data Projects: Overfitting, Grid Search and Ceiling Analysis

Updated 7 October 2026 · Fact-checked

Model fit and tuning is the step where you check whether a text-based model generalizes to new data and then improve it. Overfitting means the model learns noise. Underfitting means it is too simple. You read a fitting curve, use grid search to pick hyperparameters, and use ceiling analysis to find which pipeline step to fix.

Understand Model Fit and Tuning

A model is trained on one set of data and used on data it has never seen. The goal is generalization: good performance on new data, not just on the training set. Model fit and tuning is how you check this and improve it.

Overfitting happens when the model is too complex for the data. It learns the noise and quirks of the training set. Training error is low, but error on new data is high. Underfitting happens when the model is too simple. It misses real patterns, so error is high on both training and validation data. A model that generalizes well has low error on both and a small gap between them.

Total prediction error has three parts: bias error (from a model that is too simple), variance error (from a model too sensitive to the training sample) and base error (random noise that no model can remove). Underfit models have high bias. Overfit models have high variance. A good fit balances the two.

A fitting curve plots error against model complexity. Training error keeps falling as complexity rises. Validation error falls, hits a low point, then rises again. The low point is the best complexity. To the left you underfit. To the right you overfit. A related curve plots error against training set size or training cycles.

To fight overfitting, use regularization (penalties on complexity), cross-validation (rotating which part of the data is held out), and a larger or cleaner training set. In practice you also tune hyperparameters: settings you choose before training, such as the penalty strength. Grid search tries many hyperparameter combinations and keeps the one with the best validation score. Ceiling analysis is different. It looks at the whole pipeline, such as text cleansing, feature extraction and the model, and asks which step would gain most if it were made perfect.

Key formulas to remember

Total error decomposition: Total error = Bias error + Variance error + Base error; Base error is random noise and cannot be reduced. Tuning trades bias against variance.
Underfitting pattern: High training error and high validation error (high bias); Fix by adding complexity or better features.
Overfitting pattern: Low training error and high validation error (high variance); Fix with regularization, cross-validation, more data or fewer features.
Good fit pattern: Low training error and low validation error, small gap; Out-of-sample performance is the test of a model.
Grid search: Best hyperparameters = combination with the best validation score across the grid; Number of runs = product of the number of values tried for each hyperparameter.
Ceiling analysis: Gain of a step = pipeline accuracy with that step made perfect − current accuracy; Fix the step with the largest gain first.

How to solve Model Fit and Tuning questions

Use this order for any item-set question on model fit and tuning.

1Find the training and validation (or test) performance figures in the vignette or exhibit.
2Compare them. Both poor means underfitting. Training good but validation poor means overfitting. Both good and close means a good fit.
3Match the problem to the fix: underfitting needs more complexity or features; overfitting needs regularization, cross-validation, more data or fewer features.
4If the question lists several hyperparameter values, identify it as grid search and pick the combination with the best validation score. Multiply values to count runs.
5If the question gives accuracy as each pipeline step is made perfect, identify it as ceiling analysis. Compute each step's gain from the previous figure.
6Choose the step with the largest gain as the best place to spend effort.
7Check the wording of each option against the data given. Reject options that contradict the exhibit.

Quickest way: Two-number fit test

When to use it: When the vignette gives training and validation error or accuracy and asks you to diagnose the model.

Write the two numbers side by side.
Ask: is training poor? If yes, underfitting.
If training is good, ask: is there a big gap to validation? If yes, overfitting.
If no big gap and both good, the fit is acceptable.
For ceiling analysis, subtract each accuracy from the next one and pick the biggest jump.

Common mistakes in Model Fit and Tuning

Saying low training error means a good model.
Students trust the figure they see first.
Fix: Always check validation or test performance. Low training error with high validation error is overfitting.
Confusing bias error with variance error.
Both words sound like generic error.
Fix: Bias is from being too simple (underfit). Variance is from being too sensitive to training data (overfit).
Treating grid search as a way to find the weakest pipeline step.
Both are called tuning methods.
Fix: Grid search tunes hyperparameters of one model. Ceiling analysis compares steps of the whole pipeline.
Picking the ceiling-analysis step with the highest final accuracy instead of the largest gain.
Students read the cumulative column, not the increments.
Fix: Subtract each accuracy from the one before it. The largest increment wins.
Adding more complexity to cure overfitting.
Students assume a better fit to the training data is always better.
Fix: Overfitting needs simpler models, penalties, cross-validation or more data.
Thinking base error can be tuned away.
Students expect tuning to reach zero error.
Fix: Base error is random noise. Tuning only reduces bias and variance error.

Worked examples

Example 1

An analyst builds a model to classify earnings call transcripts as positive or negative. Accuracy is 98% on the training set and 71% on the validation set. A second, simpler model scores 64% on training and 62% on validation. Questions: (1) Diagnose each model. (2) Which fix suits the first model?

Show the solution

First model: training 98% is high, validation 71% is much lower. The gap is 27 percentage points.
A large gap with strong training results is overfitting (high variance).
Second model: both scores are low and close together, so it is underfitting (high bias).
For the overfit model, suitable fixes are regularization, cross-validation, more training data or fewer features.

Answer: (1) The first model overfits; the second underfits. (2) Apply regularization or cross-validation, or add data or remove features, for the first model.

Example 2

A team tunes a text classifier with grid search. It tries 4 values of a penalty parameter and 3 values of a learning rate, and scores each combination on validation data. Separately, ceiling analysis of the pipeline gives overall accuracy: current pipeline 70%; text cleansing made perfect 74%; feature selection also made perfect 83%; model also made perfect 86%. Questions: (1) How many models does grid search train? (2) Which step should the team improve first?

Show the solution

Grid search tries every combination: 4 × 3 = 12 runs.
Ceiling gains: cleansing 74 − 70 = 4 points.
Feature selection 83 − 74 = 9 points.
Model 86 − 83 = 3 points.
The largest gain is feature selection at 9 points.

Answer: (1) 12 combinations. (2) Improve feature selection first, since perfecting it adds 9 points.

Exam tips

Look for the training and validation figures first. Most questions on this topic can be solved from those two numbers.
Know the one-line difference: grid search tunes hyperparameters, ceiling analysis finds the pipeline step to improve.
In ceiling analysis, compute increments, not totals.
Link overfitting to variance and underfitting to bias; options often swap them.
Answer from the vignette. If the exhibit shows the data, the correct option will agree with it.

Model Fit and Tuning: frequently asked questions

What is the difference between overfitting and underfitting?

An overfit model learns noise, so it does well on training data but poorly on new data. An underfit model is too simple and does poorly on both. Overfitting is high variance. Underfitting is high bias.

What is grid search in a big data project?

Grid search tries many combinations of hyperparameter values. Each combination is scored on validation data. You keep the combination with the best score.

What is ceiling analysis?

Ceiling analysis looks at each step of a pipeline, such as text cleansing, feature selection and the model. You imagine each step made perfect and see how much overall performance improves. You then work on the step with the biggest gain.

What does a fitting curve show?

It plots error against model complexity. Training error keeps falling, while validation error falls then rises. The lowest point of validation error shows the best complexity.