Intermediate level · Module 5: Evaluating AI Outputs · Lesson 2 of 3
Automated and Model-Graded Evaluation
Automate parts of evaluation with rule-based checks and, carefully, with models acting as graders.
In this lesson you will
- Write automated checks for objective criteria
- Use model-based grading and validate it against human grades
Key concepts
rule-based checksmodel-graded evaluationagreement with humansbias in graders
The lesson
Some criteria can be checked automatically: does the answer cite a source? Is the cited source among the retrieved ones? Does an out-of-scope answer contain the referral sentence?
A language model can grade answers against a rubric, speeding up evaluation. But model graders can be biased — for example favouring longer answers — so measure their agreement with human grades on a sample before trusting them.
Use automation to run evaluations often, and humans to check samples and difficult cases.
Step by step
- Write three automated checks.
- Set up model-based grading with the rubric.
- Compare model grades with human grades on 15 answers.
- Decide how far to trust the model grader.
Practice activity
Automate the FAQ assistant's evaluation and validate the model grader.
Expected result: Automated checks plus a model grader with measured agreement.
Check your understanding
-
Give an example of an automated check for a RAG assistant.
Show answer
Checking that each answer cites a retrieved source.
-
Why validate a model grader against humans?
Show answer
Model graders can be biased or inconsistent.
Taught live by an instructor — online or in person. Free course lessons are for self-study and do not lead to a certificate — see registration options.