Advanced level · Module 1: Causal Inference and Programme Evaluation · Lesson 2 of 4
Randomised Evaluations and A/B Tests
Analyse randomised experiments correctly and understand why randomisation gives credible comparisons.
In this lesson you will
- Explain why randomisation removes confounding
- Analyse an experiment and check balance and power
Key concepts
random assignmentbalance checksstatistical powerintention to treat
The lesson
If units are assigned to the programme at random, the groups are similar on average in every way, measured or not. Differences in outcomes can then be attributed to the programme.
Check balance on baseline characteristics and analyse by original assignment (intention to treat), even if some did not take part.
Plan sample size in advance with a power calculation: too small an experiment may miss a real effect. Ethical review and consent are essential when people's services are affected.
Step by step
- Check baseline balance in a provided experiment.
- Estimate the effect with intention-to-treat analysis.
- Calculate a confidence interval for the effect.
- Run a simple power calculation.
Practice activity
Analyse the provided randomised SMS-reminder experiment on clinic attendance.
Expected result: A correct effect estimate with balance checks and interpretation.
Check your understanding
-
Why does randomisation support causal conclusions?
Show answer
It makes groups similar on average in all characteristics, measured or not.
-
What is intention-to-treat analysis?
Show answer
Analysing units by their original assignment regardless of participation.
Taught live by an instructor — online or in person. Free course lessons are for self-study and do not lead to a certificate — see registration options.