learning-it by example

#200 Machine learning and data science 6.6 min

Imbalanced Data: Class Weights vs Threshold vs Resampling

A plain model caught 8 of 39 made-up frauds; class weights, a tuned threshold and resampling each caught 30 to 34.

1. On the made-up payments, how many of the 39 test frauds did the plain logistic regression catch?
2. What did the three fixes cost, according to the video?
3. Which rows is the threshold of 0.04 picked on?
4. Why should you split the data before resampling?
5. What does class_weight set to balanced do to a fraud row here?
6. Why do the small gaps between the fixes mean little in this video?

Practice only: nothing is saved and no score counts towards anything. Each answer comes with the reason.

  1. Read which rows each step may use

    Copy the code folder first. Open run.py and write down, for each of these three jobs, which rows it uses: fitting the models, choosing the threshold, and scoring in the report calls. The answer should be training, validation and test rows, with no overlap between the three jobs.

    A hint

    Look at what is passed to fit, to pick_threshold and to report.

  2. Ask for 90% recall

    In fixes.py change target_recall=0.8 to target_recall=0.9. Predict whether the picked threshold will go up or down, and what happens to the number of false alarms on the test rows. Then run python run.py and compare the threshold line and the plain, threshold row with the video.

    A hint

    A higher recall target means catching more of the 33 validation frauds.

  3. A gentler class weight

    In run.py change LogisticRegression(class_weight="balanced") to LogisticRegression(class_weight={0: 1, 1: 10}). Predict whether the true positives and false alarms will land between the plain model and the balanced model. Run python run.py and compare the three rows.

    A hint

    Balanced made each fraud row count about 35 times as much. Here it counts 10 times as much.

  4. More fraud in the data

    In data.py change rate=0.03 to rate=0.10. Predict whether the plain model at threshold 0.5 will catch a larger or smaller share of the frauds, and whether the threshold on validation rows will change. Run python run.py and compare the first line and the plain row.

    A hint

    The first line prints how many frauds there are in each split, so check it before you compare.

These tasks are for you to try on your own machine. Nothing is checked or saved.

What this does not show

Made-up data with two columns, one model, one seed and one split, with only 39 frauds in the test set.

Up next in Machine learning and data science

More in Machine learning and data science

All Machine learning and data science videos

Related videos