#174 AI coding and productivity 5.4 min
Review AI-Generated Code Fast: Checklist vs Bandit vs Tests
A seven point checklist found seven of eight planted problems in AI style code. A security linter found three.
Run the checklist on the sample
Copy the code folder first. Run python review.py sample.py. Count the findings and note the line number of each one. Predict first how many findings there will be. When I ran it, there were 7 findings, one of them on line 28 for the import fastjsonutils.
A hint
Each finding starts with a line number and a point number from 1 to 7.
Fix the boundary bug and run the tests
Run python -m pytest -q test_sample.py and predict which test fails. Then in sample.py change qty > 10 to qty >= 10 in the discount function and run pytest again. When I ran it after the change, all 3 tests passed. Then run python review.py sample.py: it still gave 7 findings, because the checklist cannot see this bug.
A hint
The docstring says ten or more items get 10 percent off.
Silence the timeout finding
In the notify function of sample.py, add timeout=10 to the urlopen call, so it reads urlopen(request, timeout=10). Do not call the function. Predict how many findings review.py reports now, then run python review.py sample.py. When I ran it, the findings dropped from 7 to 6 and the network timeout line was gone.
A hint
Check number 6 looks for a timeout keyword on the urlopen call.
Add a test for the discount rule
Add a new test to test_sample.py that checks discount(200, 20) equals 180, in the same style as the others. Predict whether it passes after the boundary fix from the earlier exercise, then run pytest. Next, think of a rule that the discount docstring states and write one more test for it.
A hint
The docstring says ten or more items get 10 percent off, so 10 percent off 200 is 20.
These tasks are for you to try on your own machine. Nothing is checked or saved.
What this does not show
No AI model was run: the sample is hand-written with eight planted problems, and the checklist was written knowing them, so the scores show what each reviewer can see, not how often it finds problems in real code. Bandit 1.9.4 ran with default settings; its secret and timeout checks did not match this variable name (API_KEY) or this call (urllib urlopen). Only one logic bug was planted.
Up next in AI coding and productivity
More in AI coding and productivity
Related videos
- #172 Refactor Safely With AI: Characterisation Tests (pytest) AI coding and productivity
- #169 Write Coding-Assistant Rules That Work AI coding and productivity
- #171 TDD With an AI Assistant: Red, Green, Refactor (unittest) AI coding and productivity
- #177 Vibe Coding vs Engineering: Audit a Quick App in Python AI coding and productivity
