Free course · Lesson 4 of 5

Model Evaluation & Avoiding Overfitting

In this lesson you will

Read a confusion matrix
Recognise overfitting

A model that performs perfectly on training data but poorly on new data is overfitting. It has memorized the training examples instead of learning the general pattern. This is one of the most common mistakes in ML. The solution: cross-validation and regularization.

Overfitting: Model is too complex — it fits the noise in the training data. Symptoms: near-perfect training accuracy, poor test accuracy.
Underfitting: Model is too simple — it fails to capture patterns even in the training data.
Cross-Validation: Split data into K "folds," train on K-1 folds, test on the remaining fold. Repeat K times. This gives a more reliable estimate of model performance.
Regularization: Penalize model complexity to prevent overfitting (e.g., max_depth in Decision Trees, C in SVM).

💡 Practical Skill: Always compare training accuracy with test accuracy. If training is much higher, you are overfitting. Use train_test_split with random_state for reproducible results, and start with a simple model before trying complex ones.

Worked exampleDetecting Overfitting

Python

# Demonstrate overfitting vs good fit# Install: pip install scikit-learn import numpy as npfrom sklearn.tree import DecisionTreeRegressorfrom sklearn.model_selection import train_test_splitfrom sklearn.metrics import mean_squared_error # Create synthetic data with noisenp.random.seed(42)X = np.linspace(0, 10, 50).reshape(-1, 1)y = np.sin(X).flatten() + np.random.normal(0, 0.3, 50) # SplitX_train, X_test, y_train, y_test = train_test_split(    X, y, test_size=0.3, random_state=42) # Overfitted model (no depth limit — learns all the noise)overfit_model = DecisionTreeRegressor(max_depth=None)overfit_model.fit(X_train, y_train)train_pred = overfit_model.predict(X_train)test_pred = overfit_model.predict(X_test) print("=== Overfitted Model (max_depth=None) ===")print(f"Training error: {mean_squared_error(y_train, train_pred):.4f}")print(f"Test error:     {mean_squared_error(y_test, test_pred):.4f}")print("Training is much lower — model memorized the training data!") # Good model (limited depth — learns general pattern)good_model = DecisionTreeRegressor(max_depth=3)good_model.fit(X_train, y_train)train_pred2 = good_model.predict(X_train)test_pred2 = good_model.predict(X_test) print("\n=== Good Model (max_depth=3) ===")print(f"Training error: {mean_squared_error(y_train, train_pred2):.4f}")print(f"Test error:     {mean_squared_error(y_test, test_pred2):.4f}")print("Both errors are close — model generalizes well!") # Rule of thumbprint(f"\nOverfit ratio: {mean_squared_error(y_test, test_pred) / mean_squared_error(y_train, train_pred):.1f}x")print("If this ratio is > 2x, you are overfitting.")

Practice exercise

From a confusion matrix with 40 true positives, 10 false negatives, 20 false positives and 130 true negatives, calculate precision and recall and say which matters more for disease screening.

Key takeaways

Accuracy can mislead when one class is rare
Precision and recall describe different kinds of mistakes
A big gap between training and validation scores signals overfitting

In the full program Module 4: Evaluating Machine Learning Models See how this topic is taught in depth, with practical activities and an assessment.

Free self-study lesson — no certificate. For the full instructor-led program, see registration options.