learning-it by example

#113 Data engineering and pipelines 6.4 min

Parquet Explained: Why a Reader Has Less to Read

On 50,000 made-up orders, a Parquet file was 52% of the CSV, and one query needed the amount column from 1 row group of 5.

1. In the Parquet example with 50,000 made-up orders, how big was the Parquet file compared with the CSV?
2. What are the three ways a Parquet reader can do less work, as the video describes them?
3. In the example, a query asked for the amount column where order_id was 40001 or more. How many row groups could match?
4. The video counted bytes from the Parquet footer for its 9% figure. Which thing do those footer counts not measure?
5. Why does the example skip the early row groups when it looks for order ids of 40001 or more?

Practice only: nothing is saved and no score counts towards anything. Each answer comes with the reason.

  1. Predict the column sizes

    Copy the code folder first. This code needs the pyarrow package, which the video installs as pyarrow==26.0.0. Before running anything, read run.py and predict what python run.py columns prints: which two columns take the biggest share of the column data, and why. Then, if pyarrow is installed in your environment, run it and compare your answer.

    A hint

    One column has a different value on every row. The other is random pence.

  2. Change the row group size

    In data.py, the write_parquet function sets row_group_size to 10_000. Predict how many row groups python run.py size will report if you change it to 25_000, for 50,000 rows. Then predict whether the query in python run.py skip will still match 1 row group or more. Run both to check if pyarrow is installed.

    A hint

    50,000 rows divided by the group size gives the number of groups. The query wants order ids from 40001 onwards.

  3. Move the query limit

    First set row_group_size in data.py back to 10_000 if you changed it in the last exercise. The number 40001 appears in four places in run.py (lines 30, 33, 34 and 39), including the printed query line order_id >= 40001. Change all four to 20001 and predict how many row groups can match out of 5, and how many rows read_table returns. Then run python run.py skip and compare, if pyarrow is installed.

    A hint

    Row groups hold 10,000 ids each, in order: 1 to 10000, 10001 to 20000, and so on. A group can match if its largest id is at least the limit.

  4. Shuffle the ids

    This is a prediction task. The video says it only tested sorted ids. Suppose make_rows gave the order ids in random order instead of rising down the file. Write down what you expect to happen to the number of row groups that can match for order_id >= 40001, and why the min and max of each group would change. Do not trust your guess as a fact until you have run it.

    A hint

    Think about what the largest order id in each row group would be if the ids were mixed up.

These tasks are for you to try on your own machine. Nothing is checked or saved.

What this does not show

Made-up table and one pyarrow version; bytes counted from the footer, nothing timed or read from disk; tiny row groups and sorted ids chosen on purpose; CSV compared by size only, no other formats.

Up next in Data engineering and pipelines

More in Data engineering and pipelines

All Data engineering and pipelines videos

Related videos