#08 Python for data 6.1 min
Pandas vs Polars vs DuckDB on 10M Rows, Timed and Checked
Run pandas, Polars and DuckDB on the same ten million rows on one laptop, time them carefully, and check that the answers agree.
Break the fairness check
Copy the code folder first. In bench.py change run_pandas so that it returns each sum plus 1, for example by adding 1 to every value in the dictionary. Predict what the line starting 'results agree on' will print for each file. Then run make_data.py if the data files do not exist yet (it needs about half a gigabyte of disk) and run python bench.py.
A hint
The agree function only accepts a difference smaller than a penny.
Run a single time
In bench.py change runs=3 to runs=1 in median_seconds. Run python bench.py twice and compare the two sets of times. Does any ranking change between your two runs? The video lists timing a single run as a mistake, because caching and start up costs skew it.
A hint
Close other heavy programs while it runs, as the video suggests.
Try a smaller file
In make_data.py change ROWS from 10_000_000 to 1_000_000 and run it again, then run bench.py. Before you run it, write down which tool you expect to be fastest on the CSV file and on the Parquet file. Does the ranking match the ten million row ranking in the video?
A hint
The video warns that a bigger file or a different query could rank the tools differently.
Add a filter to the job
Change all three functions in bench.py to add up only the amounts above 500 for each customer. Predict whether the results will still agree, then run it. Compare the ranking with the video's plain group by and sum.
A hint
In pandas use df[df['amount'] > 500]. In Polars add .filter(pl.col('amount') > 500) after the scan. In SQL add WHERE amount > 500.
These tasks are for you to try on your own machine. Nothing is checked or saved.
What this does not show
One laptop (12 logical processors), one query (group by and sum), medians of three runs from one run of the script. Memory was not measured and the file cache was not controlled. Other queries, sizes or machines can rank the tools differently.
Up next in Python for data
More in Python for data
Related videos
- #113 Parquet Explained: Why a Reader Has Less to Read Data engineering and pipelines
- #114 Partitioning Parquet: Which Layout Opens Fewer Files? Data engineering and pipelines
- #205 Column-Level Lineage from SQL with SQLGlot Data governance
- #216 Audit Evidence From Pipeline Logs in Python (SHA-256) Data governance
