Business Analytics · Intermediate level

Intermediate level · Module 5: Testing Changes and Causal Thinking · Lesson 3 of 3

Interpreting Test Results

Read pilot results with care, considering noise, size and practical importance.

In this lesson you will

  • Compare test and comparison results
  • Judge whether a difference is large and reliable enough to act on

Key concepts

difference between groupsnoisesample sizepractical significance

The lesson

Compare the metric in the test and comparison groups: 32% of reminded customers returned versus 25% of others is a 7-point difference.

Small groups produce noisy results; a difference from 20 customers is much less reliable than one from 2,000. Repeat or extend pilots when in doubt.

Ask whether the difference is worth the cost: does the extra margin from returning customers exceed the cost of the SMS?

Step by step

  1. Calculate the metric for both groups.
  2. Calculate the difference.
  3. Consider group size and noise.
  4. Compare benefit with cost and recommend.

Practice activity

Analyse the provided pilot results and recommend whether to scale.

Expected result: A recommendation that weighs the difference, its reliability and its cost.

Check your understanding

  1. Why are results from small groups less reliable?

    Show answer

    Random variation has a bigger effect.

  2. What must a useful improvement also be?

    Show answer

    Worth its cost.

Taught live by an instructor — online or in person. Free course lessons are for self-study and do not lead to a certificate — see registration options.