Intermediate level · Module 4: Stored Logic and Data Import · Lesson 2 of 3
Finding Duplicates
Detect exact and likely duplicate records and decide how to handle them.
In this lesson you will
- Find exact duplicates with GROUP BY and HAVING
- Find likely duplicates with normalised comparisons
Key concepts
exact duplicatelikely duplicatenormalised keysreview before merge
The lesson
Exact duplicates are found by grouping on the identifying columns: GROUP BY name, village HAVING COUNT(*) > 1.
Real data has near-duplicates: 'Nyandeng Akol' and 'Nyandeng Akol', or different spellings of the same village. Compare normalised versions — lower case, single spaces — and combine several fields such as name, age and village.
Never merge likely duplicates automatically. Produce a list for staff to review, because two different people can share a name. Record decisions so they are not repeated.
Step by step
- Find exact duplicates in the imported data.
- Create normalised comparison keys.
- Find likely duplicates with the normalised keys.
- Produce a review list for staff.
Practice activity
Produce a duplicate review report for the imported beneficiaries.
Expected result: A report listing exact and likely duplicates with the evidence for each.
Check your understanding
-
How do you find exact duplicates?
Show answer
GROUP BY the identifying columns and keep groups with COUNT(*) > 1.
-
Why not merge likely duplicates automatically?
Show answer
Different people can share similar details; a person must review them.
Taught live by an instructor — online or in person. Free course lessons are for self-study and do not lead to a certificate — see registration options.