Intermediate level · Module 5: Imbalanced and Messy Real-World Data · Lesson 3 of 3
Data Drift and Realistic Validation
Recognise how changing conditions degrade models, and validate in a way that mirrors real use.
In this lesson you will
- Explain data drift and concept drift
- Use time-based validation and drift checks
Key concepts
data driftconcept drifttime-based splitmonitoring signals
The lesson
Data drift means the inputs change — new agents, new transaction patterns. Concept drift means the relationship changes — the same behaviour no longer predicts shortages, for example after a price shock or a new competitor.
If data has a time order, validate on later periods than you train on. Random splits that mix past and future make models look better than they will be.
Plan monitoring before deployment: compare the distributions of key features and the model's predictions over time, and track actual outcomes when they arrive.
Step by step
- Split the agent data by time instead of randomly.
- Compare performance with the random split.
- Simulate a shift in behaviour and measure the drop.
- List the signals you would monitor.
Practice activity
Write the monitoring plan for the cash-shortage model, with evidence from a time-based validation and drift simulation.
Expected result: A monitoring plan grounded in measured drift effects.
Check your understanding
-
What is concept drift?
Show answer
A change in the relationship between inputs and the outcome.
-
Why validate on later time periods?
Show answer
It mirrors real use, where the model predicts the future from the past.
Taught live by an instructor — online or in person. Free course lessons are for self-study and do not lead to a certificate — see registration options.