learning-it by example

Back to the video

Multi-User RAG: Filter Before You Search (Leak Test)

Same filter, same users: applied after the search it found the user's own note 18 times out of 36; applied before the search, all 36.

Video #51 ยท RAG and vector search

Steps in the video

  1. 0:00 Three filters
  2. 0:41 The idea
  3. 1:22 Setup
  4. 1:47 Thirty six notes
  5. 2:08 Three modes
  6. 2:26 What the first test counts
  7. 3:21 The leak test
  8. 3:47 What the post filter does
  9. 4:10 What this does not show
  10. 4:50 Mistakes to avoid
  11. 5:16 My take and next steps
  12. 5:46 Recap

Files

  • docs.py (47 lines)
  • leak.py (25 lines)
  • recall.py (14 lines)
  • store.py (30 lines)

Download the code (zip)

Tested with

Python 3.12.15 | ollama==0.6.3

What this does not show

36 made-up notes, 6 users and 4 bait questions, one small model (llama3.2), one run at temperature 0. The search is a plain sort, not a vector database, and the 18 of 36 comes from my design, where every user types the same question. No prompt injection was tested.

Sources

Learning IT by Example.