learning-it by example

Back to the video

Semantic Cache for LLM Apps: Pick the Threshold (Python)

With a loose threshold, a semantic cache answered 'Do you ship to Norway?' with the answer for Ireland.

Video #53 ยท RAG and vector search

Steps in the video

  1. 0:00 Do you ship to Norway?
  2. 0:19 The idea
  3. 0:53 Setup
  4. 1:10 The test questions
  5. 1:30 The look alikes
  6. 1:46 The cache
  7. 2:03 The sweep
  8. 3:36 What this does not show
  9. 4:09 Mistakes to avoid
  10. 4:31 My take and next steps
  11. 4:56 Recap

Files

  • cache.py (30 lines)
  • demo.py (22 lines)
  • shop.py (44 lines)
  • sweep.py (34 lines)

Download the code (zip)

Tested with

Python 3.12.15 | ollama==0.6.3

What this does not show

The shop is made up and I wrote and labelled every question myself: 8 stored, 16 paraphrases, 8 look alikes. One small embedding model, one data set; no chat model ran, so no time or cost saving was measured. A demonstration, not a benchmark.

Sources

Learning IT by Example.