Read Big CSVs in Small Memory: Stream and Chunk in Python
Read all at once, a 4 MB made-up CSV peaked at over 10x its size; streamed, under 1 MB.
Video #150 ยท Python for data
Steps in the video
Files
- data.py (14 lines)
- read.py (31 lines)
- run.py (39 lines)
Tested with
Python 3.12.15
What this does not show
A small made-up file and peak memory shown only as rough tracemalloc bands from one machine; no file larger than memory, no timings, and no pandas, Polars or DuckDB.
Sources
Learning IT by Example.