learning-it by example

#316 Forward deployed engineer interviews 6 min

Build an Eval Set From Customer Tickets Without Leakage

A model that only remembers scored 100% on my test set. Split another way, 40%.

1. What is group leakage in an evaluation set made from customer tickets?
2. In the run on 60 made-up tickets, what did the memoriser score on each split?
3. In the ticket eval-set demo, what does the memoriser do?
4. How does the split by wording differ from the random split?
5. The video warns against reading the 40% from the split by wording as an exact figure. Why?

Practice only: nothing is saved and no score counts towards anything. Each answer comes with the reason.

  1. Run the two splits

    Copy the code folder first. Run python run.py. Write down for each split how many test tickets have a near copy in dev and what the memoriser scores. You should see 15 and 100% for the random split, and 0 and 40% for the split by wording.

    A hint

    Two splits answer two different questions.

  2. Hold out different wordings

    In run.py change TEST_WORDINGS to the set billing-0, delivery-0, login-0 and returns-0, keeping the quotes and commas. Predict the memoriser score for the split by wording, then run python run.py. With this change the split by wording scores 0% and still has 0 near copies. What does that say about reading the 40% as one exact number?

    A hint

    The video says other choices gave a wide range of scores.

  3. Make the near-copy check stricter

    In evalset.py change limit=0.7 to limit=1.0 in near_copies. Predict how many of the 15 random-split test tickets still count as near copies, then run python run.py. With this change the count falls to 10 while the memoriser still scores 100%. Why does the score not change? Then change limit back to 0.7, because the last exercise counts near copies.

    A hint

    The extra words please or urgent change the share of words in common.

  4. Remove the copies

    In tickets.py change copies=5 to copies=1 in the make function. Predict the number of near copies in the random split and what the memoriser scores on each split, then run python run.py. With this change there are 12 tickets, 0 near copies in either split, and the memoriser scores 33% and 25%. What does that show about where the 100% came from?

    A hint

    With one copy of each wording there is nothing similar to remember.

These tasks are for you to try on your own machine. Nothing is checked or saved.

What this does not show

Sixty tickets generated from 12 templates; the memoriser is a toy. The 40 percent is for one choice of held-out wordings (in a side check other choices gave 0 to 65 percent) on 20 test tickets. Near-copies are counted at a 70 percent word overlap. Which split is right depends on what you want to measure.

Up next in Forward deployed engineer interviews

More in Forward deployed engineer interviews

All Forward deployed engineer interviews videos

Related videos