learning-it by example

Back to the video

Partitioning Parquet: Which Layout Opens Fewer Files?

On 60,000 made-up orders, a one-month query opened 1 of 12 files by month, but 300 of 300 by customer.

Video #114 ยท Data engineering and pipelines

Steps in the video

  1. 0:00 One month query: 1 of 12 files opened, or 300 of 300, depending on layout
  2. 0:32 The idea
  3. 1:17 Setup
  4. 1:32 The made-up orders
  5. 1:53 Four layouts
  6. 2:35 Count what a query opens
  7. 3:55 What this does not show
  8. 4:35 Mistakes to avoid
  9. 5:02 My take and next steps
  10. 5:34 Recap

Files

  • layouts.py (33 lines)
  • measure.py (31 lines)
  • run.py (40 lines)

Download the code (zip)

Tested with

Python 3.12.15 | pyarrow==26.0.0

What this does not show

Made-up 60,000-row table, one query, one reader (pyarrow 26.0.0); files and row groups counted, nothing timed; the by-customer layout was bigger and the reason was not tested.

Sources

Learning IT by Example.