Beginner level · Module 5: Training Neural Networks Well · Lesson 2 of 3

Learning Rates and Schedules

Find a good learning rate and adjust it during training for faster, more stable learning.

In this lesson you will

Diagnose learning rates from loss curves
Use learning-rate schedules

Key concepts

too high vs too low learning ratelearning-rate range teststep and cosine scheduleswarm-up

The lesson

The learning rate is usually the most important setting. Too high and the loss jumps around or explodes; too low and training crawls.

A learning-rate range test trains briefly while increasing the rate and shows where loss falls fastest. Start a little below that point.

Schedules lower the learning rate during training — in steps or along a smooth cosine curve — so the model makes big early progress and fine adjustments later.

Step by step

Train with three learning rates and plot the losses.
Run a simple learning-rate range test.
Add a cosine schedule.
Compare final validation accuracy.

Data and chart

Three learning rates, three very different training runs

Training loss per epoch shows learning rates that are too small, about right and far too large.

Training loss per epoch for the same model with three learning rates (illustrative)
Epoch	lr 0.0001	lr 0.01	lr 1.0
1	2.3	2.3	2.3
2	2.28	1.4	3.1
3	2.26	0.95	4.8
4	2.24	0.7	6.2
5	2.22	0.58	7.9
6	2.2	0.52	9.4

What the data shows

lr 0.0001 is too small: the loss falls by only 0.02 per epoch, so training would take far too long.

lr 0.01 is about right: a large early drop, then smaller improvements as the model settles. lr 1.0 is far too large: each step overshoots and the loss grows — training is diverging.

Recommendation

Start near 0.01 for this model, consider a schedule that lowers it later in training, and always watch the first few epochs: a rising loss means stop and reduce the learning rate.

Practice activity

Choose and justify the learning rate and schedule for the leaf classifier.

Expected result: A choice supported by loss curves and validation results.

Check your understanding

What does a learning rate that is too high look like?

Show answer
The loss jumps around or increases.
Why lower the learning rate during training?

Show answer
To allow fine adjustments after fast early progress.

Taught live by an instructor — online or in person. Free course lessons are for self-study and do not lead to a certificate — see registration options.