Intermediate level · Module 5: Evaluating AI Outputs · Lesson 1 of 3
Evaluation Sets and Rubrics
Create representative test questions and clear rubrics for judging answers.
In this lesson you will
- Build a representative evaluation set
- Write a scoring rubric
Key concepts
evaluation setcoverage of question typesrubricexpected answers
The lesson
An evaluation set should reflect real use: common questions, rare but important ones, out-of-scope questions, ambiguous phrasing and questions in the languages users will use.
A rubric defines what a good answer is: correct, complete, grounded with a citation, safe and clear. Score each criterion separately rather than giving one vague mark.
Keep the evaluation set fixed while comparing versions, and add new cases when real users reveal new problems.
Step by step
- Collect 40 questions covering all types.
- Write expected answers or criteria.
- Write a rubric with four criteria.
- Score five answers with the rubric.
Practice activity
Prepare the evaluation set and rubric for the FAQ assistant.
Expected result: A representative 40-question set with a clear rubric.
Check your understanding
-
Why score rubric criteria separately?
Show answer
It shows exactly where answers fail.
-
When should new cases be added to an evaluation set?
Show answer
When real use reveals new problems.
Taught live by an instructor — online or in person. Free course lessons are for self-study and do not lead to a certificate — see registration options.