learning-it by example

Back to the video

Seven pandas Changes Tested on 1M Rows: Speed and Memory

Seven pandas changes on 1,000,000 made-up rows: four cleared a 3x speed bar, one did not, two cut memory.

Video #138 ยท Python for data

Steps in the video

  1. 0:00 Seven pandas changes on 1,000,000 rows: four faster, two leaner
  2. 0:33 The idea
  3. 1:02 Setup
  4. 1:29 The made-up sales
  5. 1:48 Changes one and two
  6. 2:15 Changes three and four
  7. 2:45 Changes five, six and seven
  8. 3:10 How it is timed
  9. 4:22 What this does not show
  10. 5:04 Mistakes to avoid
  11. 5:36 My take and next steps
  12. 6:01 Recap

Files

  • changes.py (48 lines)
  • data.py (14 lines)
  • run.py (43 lines)

Download the code (zip)

Tested with

Python 3.12.15 | numpy==2.5.4, pandas==3.1.0

What this does not show

Made-up data, one machine, and speed shown only as pass or fail against a low bar, so your results will differ.

Sources

Learning IT by Example.