Beginner level · Module 1: Neural Network Foundations · Lesson 3 of 3
How Networks Learn: Gradient Descent
Understand how networks adjust their weights to reduce the loss, step by step.
In this lesson you will
- Explain gradient descent and the learning rate
- Describe backpropagation and run a training loop
Key concepts
gradientgradient descentbackpropagationepoch and batch
The lesson
The gradient tells us how the loss changes as each weight changes. Gradient descent moves every weight a small step in the direction that reduces the loss; the learning rate sets the step size.
Backpropagation calculates all these gradients efficiently by working backwards from the loss through the layers. PyTorch does it automatically with loss.backward().
Training repeats over the data in batches: forward pass, loss, backward pass, optimiser step. One pass over the whole data set is an epoch.
Step by step
- Write a training loop with an optimiser.
- Train the small network for several epochs.
- Plot the loss per epoch.
- Try a very large learning rate and observe the result.
Practice activity
Train a small network on a simple data set and explain the loss curve.
Expected result: A falling loss curve and a correct explanation of each step of the loop.
Check your understanding
-
What does the learning rate control?
Show answer
The size of each weight update.
-
What does loss.backward() do in PyTorch?
Show answer
Calculates the gradients of the loss with respect to the weights.
Taught live by an instructor — online or in person. Free course lessons are for self-study and do not lead to a certificate — see registration options.