#316 Forward deployed engineer interviews 6 min
Build an Eval Set From Customer Tickets Without Leakage
A model that only remembers scored 100% on my test set. Split another way, 40%.
Run the two splits
Copy the code folder first. Run python run.py. Write down for each split how many test tickets have a near copy in dev and what the memoriser scores. You should see 15 and 100% for the random split, and 0 and 40% for the split by wording.
A hint
Two splits answer two different questions.
Hold out different wordings
In run.py change TEST_WORDINGS to the set billing-0, delivery-0, login-0 and returns-0, keeping the quotes and commas. Predict the memoriser score for the split by wording, then run python run.py. With this change the split by wording scores 0% and still has 0 near copies. What does that say about reading the 40% as one exact number?
A hint
The video says other choices gave a wide range of scores.
Make the near-copy check stricter
In evalset.py change limit=0.7 to limit=1.0 in near_copies. Predict how many of the 15 random-split test tickets still count as near copies, then run python run.py. With this change the count falls to 10 while the memoriser still scores 100%. Why does the score not change? Then change limit back to 0.7, because the last exercise counts near copies.
A hint
The extra words please or urgent change the share of words in common.
Remove the copies
In tickets.py change copies=5 to copies=1 in the make function. Predict the number of near copies in the random split and what the memoriser scores on each split, then run python run.py. With this change there are 12 tickets, 0 near copies in either split, and the memoriser scores 33% and 25%. What does that show about where the 100% came from?
A hint
With one copy of each wording there is nothing similar to remember.
These tasks are for you to try on your own machine. Nothing is checked or saved.
What this does not show
Sixty tickets generated from 12 templates; the memoriser is a toy. The 40 percent is for one choice of held-out wordings (in a side check other choices gave 0 to 65 percent) on 20 test tickets. Near-copies are counted at a 70 percent word overlap. Which split is right depends on what you want to measure.
Up next in Forward deployed engineer interviews
More in Forward deployed engineer interviews
- #302 Forward Deployed Engineer Interview: What Is Tested
- #303 The FDE Decomposition Round: Vague Ask to a Plan
- #304 Scope an AI Pilot: Metric, Baseline and Stop Rule
- #305 Retry a Flaky API Without Duplicates: Idempotency Keys
- #306 Webhooks Done Right: Signatures, Retries and Replays
- #307 Schema Drift: Map a Customer's Data and Catch Changes
Related videos
- #185 Cross-Validation Done Right: The Leakage Mistake Machine learning and data science
- #06 Test Your LLM App with pytest: An Eval Harness in 40 Lines LLM engineering and evals
- #159 Feature Stores: When a Table and a Join Are Enough MLOps and deployment
- #43 Evaluate RAG: Recall, Faithfulness and a Wrong Checker RAG and vector search
