Data Science · Advanced level

Advanced level · Module 3: Working with Larger and Spatial Data · Lesson 2 of 4

SQL for Analysts

Use SQL engines such as DuckDB and SQLite to filter, join and aggregate large data quickly.

In this lesson you will

  • Query large files with DuckDB
  • Combine SQL and pandas in one workflow

Key concepts

DuckDBanalytical SQLquerying files directlySQL to DataFrame

The lesson

SQL is often the fastest way to filter and summarise large data. DuckDB runs analytical SQL directly on CSV and Parquet files on a laptop, with no database server.

Do the heavy work — joins, filters, group-bys — in SQL, then bring the smaller result into pandas for modelling and charts.

Window functions and common table expressions handle complex questions such as each household's spending trend.

Step by step

  1. Query the Parquet file with DuckDB.
  2. Aggregate transactions by area and month.
  3. Join with a households table.
  4. Load the result into pandas.

Practice activity

Answer five questions about the transaction data with SQL.

Expected result: Correct results produced efficiently from the large file.

Check your understanding

  1. What is an advantage of DuckDB for analysts?

    Show answer

    It runs fast analytical SQL on local files without a server.

  2. Where should heavy filtering and aggregation happen?

    Show answer

    In SQL, before loading smaller results into pandas.

Taught live by an instructor — online or in person. Free course lessons are for self-study and do not lead to a certificate — see registration options.