learning-it by example

#149 Python for data 6.4 min

Missing Data in pandas: Drop, Fill or Model It?

On made-up data, dropping rows moved a mean from 40,057 to 37,458; a model on age got back to 39,972.

1. When choosing how to handle missing values in a dataset, what does the missing-data video say matters most?
2. In the video's made-up survey of 1,000 incomes, older people skipped the income question more often. The true mean income was 40,057. What mean did dropping the rows with gaps give?
3. What limit does the missing-data video state about its model fill?
4. In the video's fill_model function, how is each missing income filled in?
5. When the made-up incomes were missing completely at random, what did filling the gaps with the mean do to the spread (standard deviation), compared with the true 9,656?

Practice only: nothing is saved and no score counts towards anything. Each answer comes with the reason.

  1. Predict the dropped-rows mean

    Copy the code folder first. Read data.py and fixes.py. Before running anything, predict for each of the words mcar, mar and mnar whether the drop rows mean will be close to the true mean of 40,057 or clearly below it, and say why. Then run python run.py mcar, python run.py mar and python run.py mnar, and compare the drop rows line with your predictions.

    A hint

    Ask which people are more likely to lose their income, and whether they earn more or less than average.

  2. Predict how many gaps appear

    In data.py change the 0.3 in the mcar branch of hide to 0.5. Before running, predict roughly how many of the 1,000 incomes will be reported missing, and what will happen to the spread on the fill mean line. Then run python run.py mcar and compare.

    A hint

    The first printed line shows the missing count. The spread is the standard deviation column.

  3. Fill with the median

    In fixes.py change fill_mean so it fills with df["income"].median() instead of the mean. Predict whether the spread on the fill mean line will still shrink compared with the true line, then run python run.py mcar and look at the spread column.

    A hint

    Think about what happens to the spread when about 30 percent of the rows all get the same single number.

  4. Compare rows with and without gaps

    The video's tip is to compare rows with gaps and rows without, on something you did observe. In run.py, after holed is made, add lines that print the mean age of the rows where income is missing and the mean age of the rows where it is present, using holed["income"].isna(). Predict which of mcar, mar and mnar will show a clear difference, then run all three words.

    A hint

    Use a name such as gap = holed["income"].isna(), then holed[gap]["age"].mean() and holed[~gap]["age"].mean().

These tasks are for you to try on your own machine. Nothing is checked or saved.

What this does not show

Made-up data where the true answer and the reason for each gap are known by design; one simple model, filled once, and no multiple imputation.

Up next in Python for data

More in Python for data

All Python for data videos

Related videos