#113 Data engineering and pipelines 6.4 min
Parquet Explained: Why a Reader Has Less to Read
On 50,000 made-up orders, a Parquet file was 52% of the CSV, and one query needed the amount column from 1 row group of 5.
Predict the column sizes
Copy the code folder first. This code needs the pyarrow package, which the video installs as pyarrow==26.0.0. Before running anything, read run.py and predict what python run.py columns prints: which two columns take the biggest share of the column data, and why. Then, if pyarrow is installed in your environment, run it and compare your answer.
A hint
One column has a different value on every row. The other is random pence.
Change the row group size
In data.py, the write_parquet function sets row_group_size to 10_000. Predict how many row groups python run.py size will report if you change it to 25_000, for 50,000 rows. Then predict whether the query in python run.py skip will still match 1 row group or more. Run both to check if pyarrow is installed.
A hint
50,000 rows divided by the group size gives the number of groups. The query wants order ids from 40001 onwards.
Move the query limit
First set row_group_size in data.py back to 10_000 if you changed it in the last exercise. The number 40001 appears in four places in run.py (lines 30, 33, 34 and 39), including the printed query line order_id >= 40001. Change all four to 20001 and predict how many row groups can match out of 5, and how many rows read_table returns. Then run python run.py skip and compare, if pyarrow is installed.
A hint
Row groups hold 10,000 ids each, in order: 1 to 10000, 10001 to 20000, and so on. A group can match if its largest id is at least the limit.
Shuffle the ids
This is a prediction task. The video says it only tested sorted ids. Suppose make_rows gave the order ids in random order instead of rising down the file. Write down what you expect to happen to the number of row groups that can match for order_id >= 40001, and why the min and max of each group would change. Do not trust your guess as a fact until you have run it.
A hint
Think about what the largest order id in each row group would be if the ids were mixed up.
These tasks are for you to try on your own machine. Nothing is checked or saved.
What this does not show
Made-up table and one pyarrow version; bytes counted from the footer, nothing timed or read from disk; tiny row groups and sorted ids chosen on purpose; CSV compared by size only, no other formats.
Up next in Data engineering and pipelines
More in Data engineering and pipelines
- #13 Dagster Data Pipeline with Blocking Checks: No Server Needed
- #14 dbt and DuckDB: A Full Project, Tests, and a Bad Row
- #15 Kafka Streaming Basics in Python with a Redpanda Broker
- #107 Data Contracts: Stop Breaking Downstream
- #111 Incremental Loads: Watermarks and the Late Update Trap
- #112 Idempotent Loads: Append, Upsert, Replace, Delete Insert
Related videos
- #08 Pandas vs Polars vs DuckDB on 10M Rows, Timed and Checked Python for data
- #114 Partitioning Parquet: Which Layout Opens Fewer Files? Data engineering and pipelines
- #145 Type Hints and mypy: Catch Data Bugs Before You Run Python for data
- #205 Column-Level Lineage from SQL with SQLGlot Data governance