learning-it by example

#172 AI coding and productivity 5.2 min

Refactor Safely With AI: Characterisation Tests (pytest)

A tidy refactor changed the price of every heavy express parcel. Sixty four recorded cases caught it.

1. What is a characterisation test, also called a golden master?
2. What happened when the 64 recorded pytest cases were run against the tidy refactor of the shipping code?
3. Why did the tidy refactor give a different price for heavy express parcels?
4. What do the 64 recorded cases tell you about the shipping code?
5. The video broke the old code in seven small ways. How many did three ordinary cases catch, compared with the full set of 64?

Practice only: nothing is saved and no score counts towards anything. Each answer comes with the reason.

  1. Record, then check the old code

    Copy the code folder first. Run python make_golden.py, then python -m pytest -q --tb=no, then python -m pytest -q --tb=no --impl=refactor_ai. Note how many cases are recorded and how many pass each time. On my machine the old code passed all 64 and the tidy refactor failed 8.

    A hint

    The first run writes golden.json, which the tests then read.

  2. Break the fixed refactor

    In refactor_fixed.py change the express multiplier from 1.6 to 1.5. Predict how many of the 64 cases will fail, then run python -m pytest -q --tb=no --impl=refactor_fixed. On my machine 32 failed and 32 passed. Change it back afterwards.

    A hint

    Count how many of the recorded cases are express.

  3. Add the missing error test

    Add a test to test_pricing.py that calls shipping(0, "UK", False) and expects a ValueError, using pytest.raises. Run python -m pytest -q --tb=no. Then, in legacy.py, change the check that raises for weight 0 or less so that it only raises for weight below 0, run it again, and change it back. On my machine the first run said 65 passed, and the second failed only the new test.

    A hint

    Copy the line that finds shipping from the existing test, so that --impl still works.

  4. Add an eighth breakage

    In mutants.py add an eighth entry to MUTANTS: "UK rate 2, not 1.5": ("* 1.5", "* 2"). Predict whether the three ordinary cases will catch it, then run python mutants.py. Note that the last line still says of 7, because that number is written in the script, so count the rows yourself. On my machine both the three cases and the 64 caught it.

    A hint

    Look at which weight and country the three picked cases use.

These tasks are for you to try on your own machine. Nothing is checked or saved.

What this does not show

No AI model was run: the refactor is hand-written in the style an assistant might use. The prices and the surcharge rule are made up. The 64 recorded cases say same as before, not correct, and are not every input. The seven breakages and the three-case set were picked by hand, not by a mutation testing tool.

Up next in AI coding and productivity

More in AI coding and productivity

All AI coding and productivity videos

Related videos