Missing Data in pandas: Drop, Fill or Model It?
On made-up data, dropping rows moved a mean from 40,057 to 37,458; a model on age got back to 39,972.
Video #149 ยท Python for data
Steps in the video
Files
- data.py (22 lines)
- fixes.py (16 lines)
- run.py (17 lines)
Tested with
Python 3.12.15 | numpy==2.5.4, pandas==3.1.0, python-dateutil==3.0.0, six==1.17.0, tzdata==2026.5
What this does not show
Made-up data where the true answer and the reason for each gap are known by design; one simple model, filled once, and no multiple imputation.
Sources
- https://numpy.org/doc/stable/reference/generated/numpy.polyfit.html
- https://pandas.pydata.org/docs/user_guide/missing_data.html
- https://pypi.org/pypi/numpy/json
- https://pypi.org/pypi/pandas/json
- https://pypi.org/pypi/python-dateutil/json
- https://pypi.org/pypi/six/json
- https://pypi.org/pypi/tzdata/json
- https://stefvanbuuren.name/fimd/sec-MCAR.html
- https://stefvanbuuren.name/fimd/sec-simplesolutions.html
Learning IT by Example.