Partitioning Parquet: Which Layout Opens Fewer Files?
On 60,000 made-up orders, a one-month query opened 1 of 12 files by month, but 300 of 300 by customer.
Video #114 ยท Data engineering and pipelines
Steps in the video
Files
- layouts.py (33 lines)
- measure.py (31 lines)
- run.py (40 lines)
Tested with
Python 3.12.15 | pyarrow==26.0.0
What this does not show
Made-up 60,000-row table, one query, one reader (pyarrow 26.0.0); files and row groups counted, nothing timed; the by-customer layout was bigger and the reason was not tested.
Sources
- https://arrow.apache.org/docs/python/dataset.html
- https://arrow.apache.org/docs/python/generated/pyarrow.dataset.Dataset.html
- https://arrow.apache.org/docs/python/generated/pyarrow.dataset.ParquetFileFragment.html
- https://docs.delta.io/best-practices/
- https://duckdb.org/docs/2.0/data/partitioning/hive_partitioning
- https://pypi.org/pypi/pyarrow/26.0.0/json
Learning IT by Example.