Machine Learning (ML) · Beginner level

Beginner level · Module 4: Evaluating Machine Learning Models · Lesson 3 of 3

Overfitting, Underfitting and Validation

Diagnose models that memorise or oversimplify, and use a validation set to choose model complexity.

In this lesson you will

  • Recognise overfitting and underfitting from training and validation scores
  • Use a validation set to choose a model's complexity

Key concepts

overfittingunderfittingvalidation setbias–variance trade-off

The lesson

An overfitted model performs much better on training data than on new data: it has learned noise. An underfitted model performs poorly on both: it is too simple to capture the pattern.

Use three sets: training data to fit, validation data to compare models and settings, and test data only once at the end. Choosing settings by looking at the test set turns it into another training set.

Plot training and validation scores as complexity increases, such as tree depth. The best setting is where validation performance peaks, often before training performance stops rising.

Step by step

  1. Split data into training, validation and test sets.
  2. Train trees of depth 1 to 10.
  3. Plot training and validation accuracy against depth.
  4. Choose the depth and evaluate once on the test set.

Practice activity

Produce a complexity curve for the dropout tree and justify the chosen depth.

Expected result: A curve showing where overfitting starts and a depth chosen from validation, not test, results.

Check your understanding

  1. What is the sign of overfitting?

    Show answer

    Much better performance on training data than on validation data.

  2. Why use the test set only once?

    Show answer

    Repeated use for choices makes its estimate optimistic.

Taught live by an instructor — online or in person. Free course lessons are for self-study and do not lead to a certificate — see registration options.