learning-it by example

Back to the video

Pandas vs Polars vs DuckDB on 10M Rows, Timed and Checked

Run pandas, Polars and DuckDB on the same ten million rows on one laptop, time them carefully, and check that the answers agree.

Video #08 ยท Python for data

Steps in the video

  1. 0:00 Pandas, Polars or DuckDB?
  2. 0:22 What makes a test fair
  3. 0:50 Setup
  4. 1:19 Make the data
  5. 1:55 Same job, three tools
  6. 2:30 Timing and checking
  7. 3:14 Reading the results
  8. 3:58 What this does not show
  9. 4:27 Mistakes to avoid
  10. 5:03 My take and next steps
  11. 5:32 Recap

Files

  • bench.py (48 lines)
  • make_data.py (16 lines)

Download the code (zip)

Tested with

Python 3.12.15 | pandas==3.1.0, polars==2.1.0, duckdb==2.0.0, pyarrow==26.0.0

What this does not show

One laptop (12 logical processors), one query (group by and sum), medians of three runs from one run of the script. Memory was not measured and the file cache was not controlled. Other queries, sizes or machines can rank the tools differently.

Sources

Learning IT by Example.