#148 Python for data 6.8 min
Exploratory Data Analysis in pandas: A Rerunnable Checklist
Seven pandas checks found 100 gaps, 25 repeats, 12 region spellings and 7 bad rows in made-up orders.
Predict before you run
Copy the code folder first. Read data.py and write down, before running anything, how many rows you expect from python run.py look and how many order_id repeats you expect from python run.py gaps. Then run both, with pandas and NumPy installed as in the video's requirements, and compare. The video's run printed 1825 rows and 25 repeats.
A hint
The data starts with n=1800 rows and 25 are copied to the end.
Move the price cut-off
In checks.py, in tidy, change (df["unit_price"] < 1000) to (df["unit_price"] < 30). Predict whether the number of bad rows will go up or down, and whether the number of rows left will go up or down. Then run python run.py links to check. The video's run removed 7 bad rows.
A hint
The normal prices in the data are around 40.
A stricter far-above flag
In checks.py, in values, change the 3 in q3 + 3 * (q3 - q1) to 1.5. Predict whether the count of unit_price far above the rest will stay at 2, and why. Then run python run.py values to see. The video says the 3 is the narrator's own choice.
A hint
A smaller number makes the flag catch more prices, including some ordinary ones.
Check the planted regions
In checks.py, in values, add a line to the end of the function that prints df["region"].value_counts(). Predict how many different rows the printout has, given that the check printed 12 spellings. Then run python run.py values and compare with the list of cleaned regions.
A hint
The twelve spellings come from four regions, with some lowercase and some with a trailing space.
These tasks are for you to try on your own machine. Nothing is checked or saved.
What this does not show
Made-up orders with problems I planted; no charts; cut-offs are my own choices; seven checks are not a complete analysis; pandas 3.1.0 and NumPy 2.5.4, two runs with identical output.
Up next in Python for data
More in Python for data
- #08 Pandas vs Polars vs DuckDB on 10M Rows, Timed and Checked
- #09 Clean Messy Data with Pandas 3: End to End, 3 Mistakes
- #137 Polars for pandas Users: One Report, Translated
- #138 Seven pandas Changes Tested on 1M Rows: Speed and Memory
- #139 Validate Data with Pydantic: Catch Bad Rows
- #140 Python Dates and Time Zones: 5 Bugs Reproduced
