learning-it by example

Back to the video

A/B Test Analysis in SQL and Python: z Test and Peeking

B was ahead with p 0.015; stopping at the first of ten looks raised false alarms from 4.9 to 17.2 percent.

Video #133 ยท SQL and analytics

Steps in the video

  1. 0:00 B ahead with p 0.015. Peeking at ten looks: 17.2 percent false alarms.
  2. 0:33 The idea
  3. 0:56 Setup
  4. 1:14 The made-up data
  5. 1:40 Count in SQL
  6. 2:01 The z test
  7. 2:28 The interval
  8. 3:22 The peeking trap
  9. 3:43 Two ways to decide
  10. 4:20 What this does not show
  11. 4:52 Mistakes to avoid
  12. 5:26 My take and next steps
  13. 5:53 Recap

Files

  • data.py (24 lines)
  • peek.py (27 lines)
  • run.py (24 lines)
  • ztest.py (21 lines)

Download the code (zip)

Tested with

Python 3.12.15

What this does not show

The visitors and conversions are made up; the peeking rate comes from 2,000 simulated A/A tests with ten looks of 200 and changes with the seed and the number and size of the looks.

Sources

Learning IT by Example.