learning-it by example

Back to the video

Build an Eval Set From Customer Tickets Without Leakage

A model that only remembers scored 100% on my test set. Split another way, 40%.

Video #316 ยท Forward deployed engineer interviews

Steps in the video

  1. 0:00 A model that only remembers scored 100 percent on my test set
  2. 0:30 The idea
  3. 1:06 Setup
  4. 1:32 The tickets
  5. 1:58 How similar is similar
  6. 2:17 Two splits and a check
  7. 2:35 The memoriser
  8. 2:59 Score both
  9. 3:43 What this does not show
  10. 4:32 Mistakes to avoid
  11. 5:00 My take and next steps
  12. 5:30 Recap

Files

  • evalset.py (42 lines)
  • run.py (13 lines)
  • tickets.py (29 lines)

Download the code (zip)

Tested with

Python 3.12.15

What this does not show

Sixty tickets generated from 12 templates; the memoriser is a toy. The 40 percent is for one choice of held-out wordings (in a side check other choices gave 0 to 65 percent) on 20 test tickets. Near-copies are counted at a 70 percent word overlap. Which split is right depends on what you want to measure.

Sources

Learning IT by Example.