#200 Machine learning and data science 6.6 min
Imbalanced Data: Class Weights vs Threshold vs Resampling
A plain model caught 8 of 39 made-up frauds; class weights, a tuned threshold and resampling each caught 30 to 34.
Read which rows each step may use
Copy the code folder first. Open run.py and write down, for each of these three jobs, which rows it uses: fitting the models, choosing the threshold, and scoring in the report calls. The answer should be training, validation and test rows, with no overlap between the three jobs.
A hint
Look at what is passed to fit, to pick_threshold and to report.
Ask for 90% recall
In fixes.py change target_recall=0.8 to target_recall=0.9. Predict whether the picked threshold will go up or down, and what happens to the number of false alarms on the test rows. Then run python run.py and compare the threshold line and the plain, threshold row with the video.
A hint
A higher recall target means catching more of the 33 validation frauds.
A gentler class weight
In run.py change LogisticRegression(class_weight="balanced") to LogisticRegression(class_weight={0: 1, 1: 10}). Predict whether the true positives and false alarms will land between the plain model and the balanced model. Run python run.py and compare the three rows.
A hint
Balanced made each fraud row count about 35 times as much. Here it counts 10 times as much.
More fraud in the data
In data.py change rate=0.03 to rate=0.10. Predict whether the plain model at threshold 0.5 will catch a larger or smaller share of the frauds, and whether the threshold on validation rows will change. Run python run.py and compare the first line and the plain row.
A hint
The first line prints how many frauds there are in each split, so check it before you compare.
These tasks are for you to try on your own machine. Nothing is checked or saved.
What this does not show
Made-up data with two columns, one model, one seed and one split, with only 39 frauds in the test set.
Up next in Machine learning and data science
More in Machine learning and data science
Related videos
- #183 Precision, Recall, F1: Which Metric Matters? Machine learning and data science
- #67 Distillation: Teach a Small Model From a Big One (Python) LLM engineering and evals
- #182 Linear Regression From Scratch, Then scikit-learn Machine learning and data science
- #186 Data Leakage: The Bug That Fakes Accuracy, Reproduced Machine learning and data science
