learning-it by example

Back to the video

Parquet Explained: Why a Reader Has Less to Read

On 50,000 made-up orders, a Parquet file was 52% of the CSV, and one query needed the amount column from 1 row group of 5.

Video #113 ยท Data engineering and pipelines

Steps in the video

  1. 0:00 Same orders: Parquet is 52 percent of the CSV, and one query needs 1 row group of 5
  2. 0:41 The idea
  3. 1:16 Setup
  4. 1:39 Make and write the data
  5. 2:30 Bytes per column
  6. 3:13 Skip row groups
  7. 3:34 The query
  8. 4:23 What this does not show
  9. 5:02 Mistakes to avoid
  10. 5:33 My take and next steps
  11. 5:59 Recap

Files

  • data.py (28 lines)
  • footer.py (22 lines)
  • run.py (39 lines)

Download the code (zip)

Tested with

Python 3.12.15 | pyarrow==26.0.0

What this does not show

Made-up table and one pyarrow version; bytes counted from the footer, nothing timed or read from disk; tiny row groups and sorted ids chosen on purpose; CSV compared by size only, no other formats.

Sources

Learning IT by Example.