Seven pandas Changes Tested on 1M Rows: Speed and Memory
Seven pandas changes on 1,000,000 made-up rows: four cleared a 3x speed bar, one did not, two cut memory.
Video #138 ยท Python for data
Steps in the video
- 0:00 Seven pandas changes on 1,000,000 rows: four faster, two leaner
- 0:33 The idea
- 1:02 Setup
- 1:29 The made-up sales
- 1:48 Changes one and two
- 2:15 Changes three and four
- 2:45 Changes five, six and seven
- 3:10 How it is timed
- 4:22 What this does not show
- 5:04 Mistakes to avoid
- 5:36 My take and next steps
- 6:01 Recap
Files
- changes.py (48 lines)
- data.py (14 lines)
- run.py (43 lines)
Tested with
Python 3.12.15 | numpy==2.5.4, pandas==3.1.0
What this does not show
Made-up data, one machine, and speed shown only as pass or fail against a low bar, so your results will differ.
Sources
- https://pandas.pydata.org/docs/user_guide/categorical.html
- https://pandas.pydata.org/docs/user_guide/enhancingperf.html
- https://pandas.pydata.org/docs/user_guide/groupby.html
- https://pandas.pydata.org/docs/user_guide/merging.html
- https://pandas.pydata.org/docs/user_guide/scale.html
- https://pypi.org/pypi/numpy/json
- https://pypi.org/pypi/pandas/json
Learning IT by Example.