learning-it by example

#174 AI coding and productivity 5.4 min

Review AI-Generated Code Fast: Checklist vs Bandit vs Tests

A seven point checklist found seven of eight planted problems in AI style code. A security linter found three.

1. What is the idea behind reviewing code that an AI assistant wrote, according to the video?
2. In a Python demo with eight problems planted in hand-written AI-style code, how many of the 8 planted problems did the seven point checklist, bandit and the test find?
3. A discount function's docstring says ten or more items get 10 percent off, but the code says qty > 10. Which reviewer found this boundary bug?
4. A seven point code-review script flags an import that Python cannot find. What should you do with that finding?
5. What does the video say about how far the 7 of 8 checklist score can be trusted?

Practice only: nothing is saved and no score counts towards anything. Each answer comes with the reason.

  1. Run the checklist on the sample

    Copy the code folder first. Run python review.py sample.py. Count the findings and note the line number of each one. Predict first how many findings there will be. When I ran it, there were 7 findings, one of them on line 28 for the import fastjsonutils.

    A hint

    Each finding starts with a line number and a point number from 1 to 7.

  2. Fix the boundary bug and run the tests

    Run python -m pytest -q test_sample.py and predict which test fails. Then in sample.py change qty > 10 to qty >= 10 in the discount function and run pytest again. When I ran it after the change, all 3 tests passed. Then run python review.py sample.py: it still gave 7 findings, because the checklist cannot see this bug.

    A hint

    The docstring says ten or more items get 10 percent off.

  3. Silence the timeout finding

    In the notify function of sample.py, add timeout=10 to the urlopen call, so it reads urlopen(request, timeout=10). Do not call the function. Predict how many findings review.py reports now, then run python review.py sample.py. When I ran it, the findings dropped from 7 to 6 and the network timeout line was gone.

    A hint

    Check number 6 looks for a timeout keyword on the urlopen call.

  4. Add a test for the discount rule

    Add a new test to test_sample.py that checks discount(200, 20) equals 180, in the same style as the others. Predict whether it passes after the boundary fix from the earlier exercise, then run pytest. Next, think of a rule that the discount docstring states and write one more test for it.

    A hint

    The docstring says ten or more items get 10 percent off, so 10 percent off 200 is 20.

These tasks are for you to try on your own machine. Nothing is checked or saved.

What this does not show

No AI model was run: the sample is hand-written with eight planted problems, and the checklist was written knowing them, so the scores show what each reviewer can see, not how often it finds problems in real code. Bandit 1.9.4 ran with default settings; its secret and timeout checks did not match this variable name (API_KEY) or this call (urllib urlopen). Only one logic bug was planted.

Up next in AI coding and productivity

More in AI coding and productivity

All AI coding and productivity videos

Related videos