#183 Machine learning and data science 6.5 min
Precision, Recall, F1: Which Metric Matters?
A model that never flags fraud scored 98% accuracy and caught 0 of 20 frauds.
Run the three models
Copy the code folder first. This video needs numpy and scikit-learn, as listed in requirements.txt. Predict which of the three models has the highest accuracy, then run python run.py. In the video's run, always no had the highest accuracy, 0.980, even though it caught none of the 20 frauds.
A hint
Look at the fn and rec columns for the always no row.
Try a stricter threshold
In run.py, add a model flagging scores above 0.60, for example "flag > 0.60": (score > 0.60).astype(int), to the models dictionary. Predict whether recall goes up or down compared with 0.50, then run it. In my check of the same numbers, above 0.60 gave 3 false alarms, 9 misses, accuracy 0.988, precision 0.786, recall 0.550 and a cost of 906.
A hint
A stricter threshold flags fewer payments.
Change the price of a missed fraud
In metrics.py, change missed_fraud=100 in the cost function to missed_fraud=10. Predict which of the three models now has the lowest cost, then run python run.py. In my check, the costs were 200 for always no, 106 for flag above 0.50 and 704 for flag above 0.35.
A hint
Costs are false alarms times 2 plus misses times the new price.
More frauds in the data
In data.py, change n_fraud=20 to n_fraud=40 and keep everything else. Predict what happens to the accuracy of the always no model, then run python run.py. In my check, always no had accuracy 0.960 and zero recall, and flag above 0.50 had recall 0.875.
A hint
Accuracy of always no is the share of payments that are not fraud.
These tasks are for you to try on your own machine. Nothing is checked or saved.
What this does not show
Made-up payments scored by a pretend model with two thresholds and made-up prices, compared on the same rows.
Up next in Machine learning and data science
More in Machine learning and data science
Related videos
- #200 Imbalanced Data: Class Weights vs Threshold vs Resampling Machine learning and data science
- #182 Linear Regression From Scratch, Then scikit-learn Machine learning and data science
- #186 Data Leakage: The Bug That Fakes Accuracy, Reproduced Machine learning and data science
- #187 Feature Engineering Beat 500 Trees, on Made-Up Data Machine learning and data science
