learning-it by example

Back to the video

Read Big CSVs in Small Memory: Stream and Chunk in Python

Read all at once, a 4 MB made-up CSV peaked at over 10x its size; streamed, under 1 MB.

Video #150 ยท Python for data

Steps in the video

  1. 0:00 A four megabyte CSV, read all at once, peaked at over ten times its size
  2. 0:34 The idea
  3. 1:06 Setup
  4. 1:36 The made-up orders
  5. 2:00 All at once
  6. 2:25 Streaming and chunks
  7. 2:55 Measuring the peak
  8. 3:59 What this does not show
  9. 4:32 Mistakes to avoid
  10. 5:03 My take and next steps
  11. 5:30 Recap

Files

  • data.py (14 lines)
  • read.py (31 lines)
  • run.py (39 lines)

Download the code (zip)

Tested with

Python 3.12.15

What this does not show

A small made-up file and peak memory shown only as rough tracemalloc bands from one machine; no file larger than memory, no timings, and no pandas, Polars or DuckDB.

Sources

Learning IT by Example.