#138 Python for data 6.5 min
Seven pandas Changes Tested on 1M Rows: Speed and Memory
Seven pandas changes on 1,000,000 made-up rows: four cleared a 3x speed bar, one did not, two cut memory.
Predict a failed check
Copy the code folder first. In changes.py change tax_fast so it returns df['price'] * 1.21 instead of 1.2. Predict what the 'same result' column will say for line 1 and why it matters that the run checks the answer before it looks at speed. Then run python run.py (it takes a while, because the slow versions are slow) and compare. This needs numpy and pandas as in the video's requirements.
A hint
Only line 1 uses tax_fast, so only that line should change.
Raise a bar
In run.py change the bar for change 3 from 3 to 30 in the compare call for the built-in group sum. The run prints only true or false, never seconds. Predict whether line 3 will still say True, and say what that tells you about the measured ratio. Then run python run.py to see.
A hint
The video says each bar was set far from the ratio measured in earlier runs, so a very high bar is the way to test it.
Fewer rows, smaller memory
In data.py change n=1_000_000 to n=100_000 and predict the three memory lines at the end of the output, store, customer_id and qty. Which memory numbers will change, and will the column types still shrink the same way? Run python run.py and compare with your prediction.
A hint
Memory for a column is roughly the number of rows times the size of one value.
Many unique store names
In data.py make every store name unique: use range(n) in the stores list, with a format like store_{i:07d}, and rng.permutation(n) as the index, so no name repeats. Predict what happens to the store line in the memory output, using the video's warning about categories on columns with mostly unique values. Then run python run.py and compare. When I ran it the store line showed str 62.0 MB becoming 66.0 MB as a category, so the category version was larger.
A hint
Compare the before and after memory for the store row, not the other rows.
These tasks are for you to try on your own machine. Nothing is checked or saved.
What this does not show
Made-up data, one machine, and speed shown only as pass or fail against a low bar, so your results will differ.
