learning-it by example

#150 Python for data 6 min

Read Big CSVs in Small Memory: Stream and Chunk in Python

Read all at once, a 4 MB made-up CSV peaked at over 10x its size; streamed, under 1 MB.

1. A job only needs a total per store from a large CSV file. What approach does the video suggest instead of loading every row into a list?
2. When Python's csv module summed a roughly 4 MB CSV of 200,000 made-up orders three ways, what did the tracemalloc peak memory bands show?
3. The video measures peak memory with Python's tracemalloc. What does it say about that figure?
4. In a Python csv reader that works in chunks of 10,000 rows, how are the next 10,000 rows read?
5. Which kind of job does the video say should not be chunked, because it needs all the rows together?

Practice only: nothing is saved and no score counts towards anything. Each answer comes with the reason.

  1. Make the chunks ten times bigger

    Copy the code folder first. In read.py change the size in in_chunks from 10_000 to 100_000. Predict what happens to the chunks line, then run python run.py. It takes about 40 seconds because tracing every allocation is slow. On my machine the chunks peak band moved up to under 100 MB and the over 10x flag became True, and the same totals line stayed True.

    A hint

    Bigger chunks hold more rows in memory at once.

  2. Make the chunks tiny

    In read.py change the size in in_chunks from 10_000 to 100 and run python run.py again (about 40 seconds). Predict which peak band the chunks line will print before you look. On my machine it printed under 1 MB, and the same totals line stayed True. The label still says chunks of 10,000, because it is written as text in run.py.

    A hint

    Compare with the streaming line, which reads one row at a time.

  3. Count orders instead of pence

    In totals() in read.py replace the line that computes pence with pence = 1, so each row counts as one order. Before running, predict what the total takings line will print for 200,000 rows (the script divides by 100), and whether the same totals line stays True. Then run python run.py. On my machine it printed 2,000.00 pounds and True.

    A hint

    The three readers share the same totals function, so they all change together.

  4. Stream the biggest order

    Write a new function biggest(path) in read.py that opens the CSV with csv.DictReader and finds the largest single order (qty times price_pence) while keeping only one running number, never a list. In run.py, inside the with block, compare biggest(path) with max(int(r['qty']) * int(r['price_pence']) for r in list(csv.DictReader(open(path, newline='')))) (import csv first), and check they are equal.

    A hint

    Keep a variable such as best = 0 and update it with max() for each row, as totals() keeps a running total.

These tasks are for you to try on your own machine. Nothing is checked or saved.

What this does not show

A small made-up file and peak memory shown only as rough tracemalloc bands from one machine; no file larger than memory, no timings, and no pandas, Polars or DuckDB.

Up next in Python for data

More in Python for data

All Python for data videos

Related videos