learning-it by example

#53 RAG and vector search 5.4 min

Semantic Cache for LLM Apps: Pick the Threshold (Python)

With a loose threshold, a semantic cache answered 'Do you ship to Norway?' with the answer for Ireland.

1. What does a semantic cache do that an exact match cache cannot?
2. In the video's semantic cache sweep (all-minilm embeddings, 8 stored questions, 16 paraphrases and 8 look alikes that the author wrote), at a threshold of 0.75 how many paraphrases hit the cache, and how many look alikes got through?
3. In the video's semantic cache test (all-minilm embeddings, questions the author wrote), why does the video say no single threshold catches every paraphrase and blocks every look alike?
4. In the video's SemanticCache class, when does get return a stored answer?
5. Why does the video suggest writing your look alike questions first, before picking a threshold?

Practice only: nothing is saved and no score counts towards anything. Each answer comes with the reason.

  1. Ask the cache one question

    Copy the code folder first. With Ollama running and all-minilm pulled, as in the video, open a Python prompt in the code folder. Build a SemanticCache at threshold 0.75, put all eight SEEDS into it, and call get on Do you ship to Ireland?, Do you deliver to Ireland? and Do you ship to Norway?. Predict which return an answer, then run it and print the score of each hit.

    A hint

    put takes the question, the key and the answer, in that order. When I ran it, the first two hit with scores 1.0 and 0.859 and the Norway question returned None.

  2. Move the threshold for one question

    Make a second cache the same way but at threshold 0.60, and ask Do you ship to Norway? again. Predict the answer, then run it. Compare with the cache at 0.75. What does this tell you about the cost of a loose threshold?

    A hint

    When I ran it, the 0.60 cache returned the Ireland answer with a score of 0.736.

  3. Add your own look alike

    Add a ninth look alike to LOOKALIKES in shop.py, such as Do you ship to France?, and run python sweep.py. Predict whether the highest look alike score and the false hits at 0.75 change, then run it and read the new table. Does your new question get through at any threshold in the table?

    A hint

    The video's highest look alike score was 0.736 and had no false hits at 0.75. Your new question may or may not beat that.

  4. Write a paraphrase that misses

    Add a paraphrase for stored question 2, the returns policy, that you think the cache will fail to match at 0.75, for example one with very different words. Add it to PARAPHRASES with the number 2 and run python sweep.py. Check the lowest paraphrase score line. Explain why a low score for a true paraphrase and a high score for a look alike are the problem the video describes.

    A hint

    The sweep prints the lowest paraphrase score. In the video it was 0.445, lower than the highest look alike score of 0.736.

These tasks are for you to try on your own machine. Nothing is checked or saved.

What this does not show

The shop is made up and I wrote and labelled every question myself: 8 stored, 16 paraphrases, 8 look alikes. One small embedding model, one data set; no chat model ran, so no time or cost saving was measured. A demonstration, not a benchmark.

Up next in RAG and vector search

More in RAG and vector search

All RAG and vector search videos

Related videos