Build an Eval Set From Customer Tickets Without Leakage
A model that only remembers scored 100% on my test set. Split another way, 40%.
Video #316 ยท Forward deployed engineer interviews
Steps in the video
Files
- evalset.py (42 lines)
- run.py (13 lines)
- tickets.py (29 lines)
Tested with
Python 3.12.15
What this does not show
Sixty tickets generated from 12 templates; the memoriser is a toy. The 40 percent is for one choice of held-out wordings (in a side check other choices gave 0 to 65 percent) on 20 test tickets. Near-copies are counted at a 70 percent word overlap. Which split is right depends on what you want to measure.
Sources
- http://layerwiseml.com/topics/data-leakage.html
- https://blog.dailydoseofds.com/p/when-is-random-splitting-fatal-for
- https://claveprep.com/blog/forward-deployed-engineer-system-design-interview
- https://codemia.io/knowledge-hub/path/grouped_sampling_in_scikit-learn
- https://igotanoffer.com/en/advice/openai-forward-deployed-engineer-interview
Learning IT by Example.