Machine Learning (ML) · Intermediate level

Intermediate level · Module 2 of 6 · 15 guided hours

Cross-Validation and Model Selection

Estimate performance reliably with cross-validation, compare models against baselines and choose metrics that reflect real costs.

Learning objectives

  • Use k-fold and stratified cross-validation
  • Compare several models fairly against baselines
  • Choose evaluation metrics and thresholds from the costs of errors

Topics

  • k-fold and stratified k-fold
  • cross_val_score and cross_validate
  • baselines with DummyClassifier
  • comparing models with spread
  • cost-sensitive metrics
  • precision–recall and ROC curves

Lessons

  1. Lesson 1 Cross-Validation

    Get a more reliable estimate of performance by training and evaluating on several different splits.

    k-foldstratified k-foldcross_val_scorescore variance

  2. Lesson 2 Baselines and Model Comparison

    Compare models against simple baselines and each other using the same cross-validation folds.

    baselineDummyClassifiersame folds for every modelsimplicity preference

  3. Lesson 3 Metrics and Thresholds from Real Costs

    Choose the metric and decision threshold that minimise the real cost of errors.

    precision–recall curveROC curve and AUCexpected costdecision threshold

Practical activity

Compare four candidate models for agent cash shortages with stratified cross-validation and choose one based on the expected cost of missed shortages versus unnecessary cash deliveries.

Expected outcomes

  • Reliable performance estimates
  • Model choices tied to business costs

Assessment

Model comparison report marked on correct cross-validation, baselines and cost reasoning.

Taught live by an instructor — online or in person. Free course lessons are for self-study and do not lead to a certificate — see registration options.