Deep Learning (DL) · Beginner level

Beginner level · Module 5 of 5 · 10 guided hours

Training Neural Networks Well

Choose losses, optimisers and learning rates, and prevent overfitting with regularisation, augmentation and early stopping.

Learning objectives

  • Choose suitable loss functions and optimisers
  • Set and schedule learning rates
  • Reduce overfitting with regularisation, augmentation and early stopping

Topics

  • loss functions
  • SGD and Adam
  • learning rate and schedules
  • dropout and weight decay
  • data augmentation
  • early stopping

Lessons

  1. Lesson 1 Losses and Optimisers

    Choose a loss for the task and an optimiser that updates weights efficiently.

    cross-entropy and MSEclass weights in the lossSGD with momentumAdam

  2. Lesson 2 Learning Rates and Schedules

    Find a good learning rate and adjust it during training for faster, more stable learning.

    too high vs too low learning ratelearning-rate range teststep and cosine scheduleswarm-up

  3. Lesson 3 Regularisation, Augmentation and Early Stopping

    Prevent overfitting so the model works on new photos, not just the training set.

    dropoutweight decaydata augmentationearly stopping

Practical activity

Improve the leaf disease classifier's training with a learning-rate experiment, augmentation and early stopping, and document the effect of each change.

Expected outcomes

  • Stable, well-regularised training
  • Evidence-based training choices

Assessment

Training experiment report marked on method, evidence and diagnosis.

Free taster lesson Model Optimization: Loss Functions, Optimizers & Learning Rate Read a free self-study lesson on this topic in the Free Course Library.

Taught live by an instructor — online or in person. Free course lessons are for self-study and do not lead to a certificate — see registration options.