Artificial Intelligence (AI) · Intermediate level

Intermediate level · Module 5 of 6 · 13 guided hours

Evaluating AI Outputs

Evaluate AI systems systematically with test sets, rubrics, automated checks and human review.

Learning objectives

  • Build evaluation sets and rubrics
  • Combine automated checks with careful use of model-based grading
  • Run human evaluation and error analysis

Topics

  • evaluation sets
  • rubrics and scoring
  • automated checks
  • model-graded evaluation and its limits
  • human evaluation
  • error analysis and dashboards

Lessons

  1. Lesson 1 Evaluation Sets and Rubrics

    Create representative test questions and clear rubrics for judging answers.

    evaluation setcoverage of question typesrubricexpected answers

  2. Lesson 2 Automated and Model-Graded Evaluation

    Automate parts of evaluation with rule-based checks and, carefully, with models acting as graders.

    rule-based checksmodel-graded evaluationagreement with humansbias in graders

  3. Lesson 3 Human Evaluation and Error Analysis

    Review outputs with people, categorise failures and decide what to fix.

    human reviewerror categoriesseverityevaluation reporting

Practical activity

Evaluate the FAQ assistant on 40 questions with a rubric, combining automated checks, model grading and human review, and report the results.

Expected outcomes

  • Trustworthy evaluation of AI systems
  • Clear evidence for improvement decisions

Assessment

Evaluation report marked on method, reliability and insight.

Taught live by an instructor — online or in person. Free course lessons are for self-study and do not lead to a certificate — see registration options.