Parquet Explained: Why a Reader Has Less to Read
On 50,000 made-up orders, a Parquet file was 52% of the CSV, and one query needed the amount column from 1 row group of 5.
Video #113 ยท Data engineering and pipelines
Steps in the video
Files
- data.py (28 lines)
- footer.py (22 lines)
- run.py (39 lines)
Tested with
Python 3.12.15 | pyarrow==26.0.0
What this does not show
Made-up table and one pyarrow version; bytes counted from the footer, nothing timed or read from disk; tiny row groups and sorted ids chosen on purpose; CSV compared by size only, no other formats.
Sources
Learning IT by Example.