#51 RAG and vector search 6.2 min
Multi-User RAG: Filter Before You Search (Leak Test)
Same filter, same users: applied after the search it found the user's own note 18 times out of 36; applied before the search, all 36.
Read the notes
Copy the code folder first. Open a Python prompt in it and import DOCS and USERS from docs. Print len(DOCS), then the text of the first note and the last note. Predict how many notes each user has, and check it by counting the notes whose owner is Carol.
A hint
There are 6 users and 6 topics. Use a list comprehension over DOCS.
Change the number of results
In recall.py the store returns the top 3 notes, set in retrieve with k=3. Predict how many of the 36 questions find the user's own note with the post filter if k is 6, and how that changes the number of other users' notes shown with no filter. Then run it by passing k=6 in the retrieve call. It needs Ollama with all-minilm, as in the video.
A hint
When I ran it, with k equal to 6 the post filter found 36 of 36, and the none mode showed 180 other users' notes. At k equal to 1 the post filter found only 6.
Which three notes come back
Use retrieve with the none mode for the question Where am I going on holiday? asked by Alice, then by Frank, and print the owner and topic of each returned note. Predict whether the two lists will be the same, then run it. Say in one sentence why this explains the post filter result.
A hint
Embed the question with embed from store and pass it as query=. When I ran it both lists were Alice, Eve and Bob holiday notes.
Write your own bait question
Add a fifth entry to BAIT in docs.py: a user asks about another user's gym membership, with a detail only that note holds, such as the monthly price. Predict whether the none mode and the pre mode will repeat that detail. If Ollama has llama3.2, run python leak.py to check, and note that without a graphics card the model may take several minutes. One question tells you little, so say what else you would test.
A hint
The video printed that none mode leaked 4 of 4 and pre mode 0 of 4 on its own questions. Your result is your own test.
These tasks are for you to try on your own machine. Nothing is checked or saved.
What this does not show
36 made-up notes, 6 users and 4 bait questions, one small model (llama3.2), one run at temperature 0. The search is a plain sort, not a vector database, and the 18 of 36 comes from my design, where every user types the same question. No prompt injection was tested.
Up next in RAG and vector search
More in RAG and vector search
- #02 RAG From Scratch in Python: Chunk, Embed, Retrieve (Local)
- #03 pgvector vs Chroma vs Qdrant: Same Data, Real Results
- #40 Chunking Strategies Compared on One Document
- #41 Hybrid Search: BM25 Plus Vectors, Tested
- #42 Reranking With a Cross-Encoder: Does It Help? Python Test
- #43 Evaluate RAG: Recall, Faithfulness and a Wrong Checker
Related videos
- #26 Plan, Act, Reflect: Does It Help a Small Model? AI agents and MCP
- #57 LLM as Judge: I Swapped the Order and It Changed LLM engineering and evals
- #58 JSON Mode vs Schema vs Function Calling: Measured (Ollama) LLM engineering and evals
- #63 Few-Shot Examples: How Many Is Enough? A Measured Test LLM engineering and evals
