A/B Test Analysis in SQL and Python: z Test and Peeking
B was ahead with p 0.015; stopping at the first of ten looks raised false alarms from 4.9 to 17.2 percent.
Video #133 ยท SQL and analytics
Steps in the video
Files
- data.py (24 lines)
- peek.py (27 lines)
- run.py (24 lines)
- ztest.py (21 lines)
Tested with
Python 3.12.15
What this does not show
The visitors and conversions are made up; the peeking rate comes from 2,000 simulated A/A tests with ten looks of 200 and changes with the seed and the number and size of the looks.
Sources
Learning IT by Example.