learning-it by example

Back to the video

Precision, Recall, F1: Which Metric Matters?

A model that never flags fraud scored 98% accuracy and caught 0 of 20 frauds.

Video #183 ยท Machine learning and data science

Steps in the video

  1. 0:00 98% accurate, and it caught zero fraud
  2. 0:30 The idea
  3. 1:09 Setup
  4. 1:39 The pretend model
  5. 2:04 The four boxes
  6. 2:27 The scores, and a price
  7. 3:00 Three models
  8. 3:19 Check with scikit-learn
  9. 4:26 What this does not show
  10. 4:57 Mistakes to avoid
  11. 5:26 My take and next steps
  12. 6:03 Recap

Files

  • data.py (12 lines)
  • metrics.py (20 lines)
  • run.py (26 lines)

Download the code (zip)

Tested with

Python 3.12.15 | numpy==2.5.4, scikit-learn==1.9.1, scipy==1.18.1, joblib==1.6.0, narwhals==2.27.1, threadpoolctl==3.7.0, cloudpickle==3.1.2

What this does not show

Made-up payments scored by a pretend model with two thresholds and made-up prices, compared on the same rows.

Sources

Learning IT by Example.