Pandas vs Polars vs DuckDB on 10M Rows, Timed and Checked
Run pandas, Polars and DuckDB on the same ten million rows on one laptop, time them carefully, and check that the answers agree.
Video #08 ยท Python for data
Steps in the video
Files
- bench.py (48 lines)
- make_data.py (16 lines)
Tested with
Python 3.12.15 | pandas==3.1.0, polars==2.1.0, duckdb==2.0.0, pyarrow==26.0.0
What this does not show
One laptop (12 logical processors), one query (group by and sum), medians of three runs from one run of the script. Memory was not measured and the file cache was not controlled. Other queries, sizes or machines can rank the tools differently.
Sources
Learning IT by Example.