learning-it by example

#133 SQL and analytics 6.4 min

A/B Test Analysis in SQL and Python: z Test and Peeking

B was ahead with p 0.015; stopping at the first of ten looks raised false alarms from 4.9 to 17.2 percent.

1. In the A/B test analysis video, 2,000 made-up A/A tests were scored two ways. What happened to the false alarm rate when the analyst stopped at the first of ten looks that won?
2. A had 400 conversions from 4,000 visitors and B had 468 from 4,000. What did the two-sided two-proportion z test give?
3. The z test for two proportions is only an approximation. What does the video say about when it is safe to use?
4. In the SQL query that counts an A/B test, conversions are stored as 1 or 0 in a column called converted. How does the query get the number of conversions per variant?
5. When the video builds the 95 percent interval for B minus A, which standard error does it use?

Practice only: nothing is saved and no score counts towards anything. Each answer comes with the reason.

  1. Make B's gain smaller

    Copy the code folder first. In data.py change B's conversions from 468 to 440, then run python run.py test. Predict first: will the p value get bigger or smaller, and will the 95 percent interval still leave out zero? When I ran it, B showed 11.0%, z = 1.46, p = 0.145 and an interval of -0.34% to +2.34%.

    A hint

    Only the second number in the B entry of ARMS changes.

  2. Fewer looks, fewer false alarms

    In peek.py change LOOKS from 10 to 5 and run python run.py peek. Predict whether the 'stop at any look' false alarm rate goes up or down from 17.2 percent. When I ran it, it came out at 11.6 percent, and the one-look-at-the-end figure was 4.1 percent.

    A hint

    The end figure moves a little because a different number of random draws is used.

  3. One look only

    Set LOOKS to 1 in peek.py and run python run.py peek. Predict how the two false alarm rates compare when there is only one look. When I ran it, both lines printed 4.5%, because stopping at the first win and looking at the end are the same plan.

    A hint

    With one look, the smallest p value and the last p value are the same number.

  4. Change the seed

    In peek.py the seed is 133 in false_alarms(seed=133). Change it to another number such as 7, run python run.py peek and compare with 4.9% and 17.2%. Predict how far the figures move. When I ran seed 7 I got 5.1% and 19.2%. The video says two thousand tests is a sample and another seed moves the rate by a few points.

    A hint

    Run it with two or three different seeds and note the range.

These tasks are for you to try on your own machine. Nothing is checked or saved.

What this does not show

The visitors and conversions are made up; the peeking rate comes from 2,000 simulated A/A tests with ten looks of 200 and changes with the seed and the number and size of the looks.

Up next in SQL and analytics

More in SQL and analytics

All SQL and analytics videos

Related videos