learning-it by example

#183 Machine learning and data science 6.5 min

Precision, Recall, F1: Which Metric Matters?

A model that never flags fraud scored 98% accuracy and caught 0 of 20 frauds.

1. Why can accuracy alone mislead when you score a yes or no model, such as one that flags fraud?
2. In the made-up payments data, 20 of 1000 payments are fraud. What did the model that always says no score, and how many frauds did it catch?
3. When the threshold was lowered from flagging scores above 0.50 to above 0.35, what happened to recall and precision?
4. In the fraud metrics code (metrics.py), how is precision reported when the model flags nothing, which is zero over zero?
5. The flag-above-0.50 model had the lowest cost in the video. What does the video say about that result?

Practice only: nothing is saved and no score counts towards anything. Each answer comes with the reason.

  1. Run the three models

    Copy the code folder first. This video needs numpy and scikit-learn, as listed in requirements.txt. Predict which of the three models has the highest accuracy, then run python run.py. In the video's run, always no had the highest accuracy, 0.980, even though it caught none of the 20 frauds.

    A hint

    Look at the fn and rec columns for the always no row.

  2. Try a stricter threshold

    In run.py, add a model flagging scores above 0.60, for example "flag > 0.60": (score > 0.60).astype(int), to the models dictionary. Predict whether recall goes up or down compared with 0.50, then run it. In my check of the same numbers, above 0.60 gave 3 false alarms, 9 misses, accuracy 0.988, precision 0.786, recall 0.550 and a cost of 906.

    A hint

    A stricter threshold flags fewer payments.

  3. Change the price of a missed fraud

    In metrics.py, change missed_fraud=100 in the cost function to missed_fraud=10. Predict which of the three models now has the lowest cost, then run python run.py. In my check, the costs were 200 for always no, 106 for flag above 0.50 and 704 for flag above 0.35.

    A hint

    Costs are false alarms times 2 plus misses times the new price.

  4. More frauds in the data

    In data.py, change n_fraud=20 to n_fraud=40 and keep everything else. Predict what happens to the accuracy of the always no model, then run python run.py. In my check, always no had accuracy 0.960 and zero recall, and flag above 0.50 had recall 0.875.

    A hint

    Accuracy of always no is the share of payments that are not fraud.

These tasks are for you to try on your own machine. Nothing is checked or saved.

What this does not show

Made-up payments scored by a pretend model with two thresholds and made-up prices, compared on the same rows.

Up next in Machine learning and data science

More in Machine learning and data science

All Machine learning and data science videos

Related videos