Semantic Cache for LLM Apps: Pick the Threshold (Python)
With a loose threshold, a semantic cache answered 'Do you ship to Norway?' with the answer for Ireland.
Video #53 ยท RAG and vector search
Steps in the video
Files
- cache.py (30 lines)
- demo.py (22 lines)
- shop.py (44 lines)
- sweep.py (34 lines)
Tested with
Python 3.12.15 | ollama==0.6.3
What this does not show
The shop is made up and I wrote and labelled every question myself: 8 stored, 16 paraphrases, 8 look alikes. One small embedding model, one data set; no chat model ran, so no time or cost saving was measured. A demonstration, not a benchmark.
Sources
Learning IT by Example.