learning-it by example

#138 Python for data 6.5 min

Seven pandas Changes Tested on 1M Rows: Speed and Memory

Seven pandas changes on 1,000,000 made-up rows: four cleared a 3x speed bar, one did not, two cut memory.

1. In the seven pandas changes tested on 1,000,000 made-up rows, which change did not clear its speed bar?
2. On the million made-up rows, what happened to the memory of the store column (50 repeated names) when it was changed to a category?
3. Why is turning a column of mostly unique text values into a category a mistake, according to the pandas guide in the video?
4. Why is multiplying a whole price column by 1.2 much faster than price.apply with a lambda that multiplies each price?
5. Why is calling pd.concat inside a loop for each of 100 chunks slow, and what does the pandas guide suggest instead?

Practice only: nothing is saved and no score counts towards anything. Each answer comes with the reason.

  1. Predict a failed check

    Copy the code folder first. In changes.py change tax_fast so it returns df['price'] * 1.21 instead of 1.2. Predict what the 'same result' column will say for line 1 and why it matters that the run checks the answer before it looks at speed. Then run python run.py (it takes a while, because the slow versions are slow) and compare. This needs numpy and pandas as in the video's requirements.

    A hint

    Only line 1 uses tax_fast, so only that line should change.

  2. Raise a bar

    In run.py change the bar for change 3 from 3 to 30 in the compare call for the built-in group sum. The run prints only true or false, never seconds. Predict whether line 3 will still say True, and say what that tells you about the measured ratio. Then run python run.py to see.

    A hint

    The video says each bar was set far from the ratio measured in earlier runs, so a very high bar is the way to test it.

  3. Fewer rows, smaller memory

    In data.py change n=1_000_000 to n=100_000 and predict the three memory lines at the end of the output, store, customer_id and qty. Which memory numbers will change, and will the column types still shrink the same way? Run python run.py and compare with your prediction.

    A hint

    Memory for a column is roughly the number of rows times the size of one value.

  4. Many unique store names

    In data.py make every store name unique: use range(n) in the stores list, with a format like store_{i:07d}, and rng.permutation(n) as the index, so no name repeats. Predict what happens to the store line in the memory output, using the video's warning about categories on columns with mostly unique values. Then run python run.py and compare. When I ran it the store line showed str 62.0 MB becoming 66.0 MB as a category, so the category version was larger.

    A hint

    Compare the before and after memory for the store row, not the other rows.

These tasks are for you to try on your own machine. Nothing is checked or saved.

What this does not show

Made-up data, one machine, and speed shown only as pass or fail against a low bar, so your results will differ.

Up next in Python for data

More in Python for data

All Python for data videos

Related videos