#53 RAG and vector search 5.4 min
Semantic Cache for LLM Apps: Pick the Threshold (Python)
With a loose threshold, a semantic cache answered 'Do you ship to Norway?' with the answer for Ireland.
Ask the cache one question
Copy the code folder first. With Ollama running and all-minilm pulled, as in the video, open a Python prompt in the code folder. Build a SemanticCache at threshold 0.75, put all eight SEEDS into it, and call get on Do you ship to Ireland?, Do you deliver to Ireland? and Do you ship to Norway?. Predict which return an answer, then run it and print the score of each hit.
A hint
put takes the question, the key and the answer, in that order. When I ran it, the first two hit with scores 1.0 and 0.859 and the Norway question returned None.
Move the threshold for one question
Make a second cache the same way but at threshold 0.60, and ask Do you ship to Norway? again. Predict the answer, then run it. Compare with the cache at 0.75. What does this tell you about the cost of a loose threshold?
A hint
When I ran it, the 0.60 cache returned the Ireland answer with a score of 0.736.
Add your own look alike
Add a ninth look alike to LOOKALIKES in shop.py, such as Do you ship to France?, and run python sweep.py. Predict whether the highest look alike score and the false hits at 0.75 change, then run it and read the new table. Does your new question get through at any threshold in the table?
A hint
The video's highest look alike score was 0.736 and had no false hits at 0.75. Your new question may or may not beat that.
Write a paraphrase that misses
Add a paraphrase for stored question 2, the returns policy, that you think the cache will fail to match at 0.75, for example one with very different words. Add it to PARAPHRASES with the number 2 and run python sweep.py. Check the lowest paraphrase score line. Explain why a low score for a true paraphrase and a high score for a look alike are the problem the video describes.
A hint
The sweep prints the lowest paraphrase score. In the video it was 0.445, lower than the highest look alike score of 0.736.
These tasks are for you to try on your own machine. Nothing is checked or saved.
What this does not show
The shop is made up and I wrote and labelled every question myself: 8 stored, 16 paraphrases, 8 look alikes. One small embedding model, one data set; no chat model ran, so no time or cost saving was measured. A demonstration, not a benchmark.
Up next in RAG and vector search
More in RAG and vector search
- #02 RAG From Scratch in Python: Chunk, Embed, Retrieve (Local)
- #03 pgvector vs Chroma vs Qdrant: Same Data, Real Results
- #40 Chunking Strategies Compared on One Document
- #41 Hybrid Search: BM25 Plus Vectors, Tested
- #42 Reranking With a Cross-Encoder: Does It Help? Python Test
- #43 Evaluate RAG: Recall, Faithfulness and a Wrong Checker
Related videos
- #69 Detect LLM Hallucinations: 5 Checks Scored on 16 Answers LLM engineering and evals
- #293 RAG for Beginners: The Idea, Then 37 Lines of Python RAG and vector search
- #46 GraphRAG Explained With a Tiny Graph in Python RAG and vector search
- #52 RAG Citations: Source Removed, Exact Quote, Wrong Source RAG and vector search
