#133 SQL and analytics 6.4 min
A/B Test Analysis in SQL and Python: z Test and Peeking
B was ahead with p 0.015; stopping at the first of ten looks raised false alarms from 4.9 to 17.2 percent.
Make B's gain smaller
Copy the code folder first. In data.py change B's conversions from 468 to 440, then run python run.py test. Predict first: will the p value get bigger or smaller, and will the 95 percent interval still leave out zero? When I ran it, B showed 11.0%, z = 1.46, p = 0.145 and an interval of -0.34% to +2.34%.
A hint
Only the second number in the B entry of ARMS changes.
Fewer looks, fewer false alarms
In peek.py change LOOKS from 10 to 5 and run python run.py peek. Predict whether the 'stop at any look' false alarm rate goes up or down from 17.2 percent. When I ran it, it came out at 11.6 percent, and the one-look-at-the-end figure was 4.1 percent.
A hint
The end figure moves a little because a different number of random draws is used.
One look only
Set LOOKS to 1 in peek.py and run python run.py peek. Predict how the two false alarm rates compare when there is only one look. When I ran it, both lines printed 4.5%, because stopping at the first win and looking at the end are the same plan.
A hint
With one look, the smallest p value and the last p value are the same number.
Change the seed
In peek.py the seed is 133 in false_alarms(seed=133). Change it to another number such as 7, run python run.py peek and compare with 4.9% and 17.2%. Predict how far the figures move. When I ran seed 7 I got 5.1% and 19.2%. The video says two thousand tests is a sample and another seed moves the rate by a few points.
A hint
Run it with two or three different seeds and note the range.
These tasks are for you to try on your own machine. Nothing is checked or saved.
What this does not show
The visitors and conversions are made up; the peeking rate comes from 2,000 simulated A/A tests with ten looks of 200 and changes with the seed and the number and size of the looks.
Up next in SQL and analytics
More in SQL and analytics
Related videos
- #157 Model Registry by Hand: Versions, Promote, Roll Back MLOps and deployment
- #208 Right to Erasure Across a Pipeline: Delete, Scan, Restore Data governance
- #238 Event-Driven AI Agents: Queues, Crashes and Duplicates Enterprise architecture and AI agents
- #246 Data Architecture for AI Agents: Tables, Vectors, Graphs Enterprise architecture and AI agents
