learning-it by example

#148 Python for data 6.8 min

Exploratory Data Analysis in pandas: A Rerunnable Checklist

Seven pandas checks found 100 gaps, 25 repeats, 12 region spellings and 7 bad rows in made-up orders.

1. The pandas EDA checklist ran on 1,825 made-up orders. After dropping the repeated rows and the bad rows, how many rows were left, and how many of each were removed?
2. The units column holds whole numbers but pandas reported float64. What reason does the video give?
3. The check found 25 exact duplicate rows and 25 repeated order ids. What does the video say that suggests?
4. The 'delivery_days below 1' flag counted 3 rows. Why did the two delivery times of 400 not show up in that flag?
5. After tidying, the mean units rose from 2.95 at no discount to 5.11 at a discount of 0.20, with a correlation of 0.42. How does the video say to read this?

Practice only: nothing is saved and no score counts towards anything. Each answer comes with the reason.

  1. Predict before you run

    Copy the code folder first. Read data.py and write down, before running anything, how many rows you expect from python run.py look and how many order_id repeats you expect from python run.py gaps. Then run both, with pandas and NumPy installed as in the video's requirements, and compare. The video's run printed 1825 rows and 25 repeats.

    A hint

    The data starts with n=1800 rows and 25 are copied to the end.

  2. Move the price cut-off

    In checks.py, in tidy, change (df["unit_price"] < 1000) to (df["unit_price"] < 30). Predict whether the number of bad rows will go up or down, and whether the number of rows left will go up or down. Then run python run.py links to check. The video's run removed 7 bad rows.

    A hint

    The normal prices in the data are around 40.

  3. A stricter far-above flag

    In checks.py, in values, change the 3 in q3 + 3 * (q3 - q1) to 1.5. Predict whether the count of unit_price far above the rest will stay at 2, and why. Then run python run.py values to see. The video says the 3 is the narrator's own choice.

    A hint

    A smaller number makes the flag catch more prices, including some ordinary ones.

  4. Check the planted regions

    In checks.py, in values, add a line to the end of the function that prints df["region"].value_counts(). Predict how many different rows the printout has, given that the check printed 12 spellings. Then run python run.py values and compare with the list of cleaned regions.

    A hint

    The twelve spellings come from four regions, with some lowercase and some with a trailing space.

These tasks are for you to try on your own machine. Nothing is checked or saved.

What this does not show

Made-up orders with problems I planted; no charts; cut-offs are my own choices; seven checks are not a complete analysis; pandas 3.1.0 and NumPy 2.5.4, two runs with identical output.

Up next in Python for data

More in Python for data

All Python for data videos

Related videos