Advanced level · Module 1 of 6 · 24 guided hours
Attention and Transformers
Understand attention and the transformer architecture, and fine-tune pre-trained transformers for language and vision tasks.
Learning objectives
- Explain self-attention and the transformer architecture
- Use pre-trained transformer models and tokenisers
- Fine-tune a transformer for classification and evaluate it
Topics
- attention and self-attention
- queries, keys and values
- transformer encoder and decoder
- positional information
- pre-trained models and tokenisers
- fine-tuning and vision transformers
Lessons
-
Lesson 1 Attention
Understand how attention lets a model weigh every part of its input when producing each output.
attention weightsqueries, keys and valuessoftmaxself-attention
-
Lesson 2 The Transformer Architecture
See how attention, feed-forward layers, residual connections and positional information form a transformer.
encoder and decodermulti-head attentionpositional encodinglayer normalisation
-
Lesson 3 Pre-trained Models and Tokenisers
Load pre-trained transformers and their tokenisers, and understand what they can and cannot handle.
model hubtokenisermultilingual modelsmodel licences
-
Lesson 4 Fine-Tuning Transformers
Fine-tune a transformer for classification, evaluate it and compare it with simpler models.
classification headfine-tuning settingsevaluationvision transformers
Practical activity
Fine-tune a small pre-trained multilingual transformer to classify synthetic SMS feedback messages into programme topics, and compare it with the LSTM from the Beginner level.
Expected outcomes
- Working knowledge of transformers
- Fine-tuned models for real tasks
Assessment
Practical: fine-tune and evaluate a transformer on a provided task; marked on correctness and comparison.
Taught live by an instructor — online or in person. Free course lessons are for self-study and do not lead to a certificate — see registration options.