Advanced level · Module 1: Attention and Transformers · Lesson 4 of 4
Fine-Tuning Transformers
Fine-tune a transformer for classification, evaluate it and compare it with simpler models.
In this lesson you will
- Fine-tune a pre-trained transformer for classification
- Compare it with simpler baselines fairly
Key concepts
classification headfine-tuning settingsevaluationvision transformers
The lesson
Fine-tuning adds a classification head and trains the model briefly on labelled examples, with a small learning rate and few epochs.
Compare with simpler baselines — a keyword model, the LSTM — on the same test set. Transformers usually win on varied language, but the gain must justify their size and cost.
The same approach applies to vision transformers, which split images into patches treated like tokens and are strong when pre-trained on large data.
Step by step
- Prepare the SMS data with the tokeniser.
- Fine-tune the model for a few epochs.
- Evaluate per class on the test set.
- Compare with the LSTM and a keyword baseline.
Practice activity
Complete the SMS topic classification practical activity.
Expected result: A fine-tuned model with a fair comparison and per-class results.
Check your understanding
-
Which settings suit transformer fine-tuning?
Show answer
A small learning rate and few epochs.
-
How do vision transformers treat images?
Show answer
As sequences of image patches, similar to tokens.
Taught live by an instructor — online or in person. Free course lessons are for self-study and do not lead to a certificate — see registration options.