learning-it by example

#51 RAG and vector search 6.2 min

Multi-User RAG: Filter Before You Search (Leak Test)

Same filter, same users: applied after the search it found the user's own note 18 times out of 36; applied before the search, all 36.

1. In a multi-user RAG app, what is the difference between a post filter and a pre filter on the owner field?
2. In the video's multi-user RAG test (36 made-up notes, six users, top three results, all-minilm embeddings), the post filter showed no other user's note. How many of the 36 questions found the user's own note?
3. In the video's multi-user RAG store, how does pre mode differ from post mode when picking the top three notes?
4. In the video's multi-user RAG test, why did the post filter find the user's own note exactly 18 times out of 36, and what does the video say about that number?
5. Where should the user id used for the filter come from in a real app?

Practice only: nothing is saved and no score counts towards anything. Each answer comes with the reason.

  1. Read the notes

    Copy the code folder first. Open a Python prompt in it and import DOCS and USERS from docs. Print len(DOCS), then the text of the first note and the last note. Predict how many notes each user has, and check it by counting the notes whose owner is Carol.

    A hint

    There are 6 users and 6 topics. Use a list comprehension over DOCS.

  2. Change the number of results

    In recall.py the store returns the top 3 notes, set in retrieve with k=3. Predict how many of the 36 questions find the user's own note with the post filter if k is 6, and how that changes the number of other users' notes shown with no filter. Then run it by passing k=6 in the retrieve call. It needs Ollama with all-minilm, as in the video.

    A hint

    When I ran it, with k equal to 6 the post filter found 36 of 36, and the none mode showed 180 other users' notes. At k equal to 1 the post filter found only 6.

  3. Which three notes come back

    Use retrieve with the none mode for the question Where am I going on holiday? asked by Alice, then by Frank, and print the owner and topic of each returned note. Predict whether the two lists will be the same, then run it. Say in one sentence why this explains the post filter result.

    A hint

    Embed the question with embed from store and pass it as query=. When I ran it both lists were Alice, Eve and Bob holiday notes.

  4. Write your own bait question

    Add a fifth entry to BAIT in docs.py: a user asks about another user's gym membership, with a detail only that note holds, such as the monthly price. Predict whether the none mode and the pre mode will repeat that detail. If Ollama has llama3.2, run python leak.py to check, and note that without a graphics card the model may take several minutes. One question tells you little, so say what else you would test.

    A hint

    The video printed that none mode leaked 4 of 4 and pre mode 0 of 4 on its own questions. Your result is your own test.

These tasks are for you to try on your own machine. Nothing is checked or saved.

What this does not show

36 made-up notes, 6 users and 4 bait questions, one small model (llama3.2), one run at temperature 0. The search is a plain sort, not a vector database, and the 18 of 36 comes from my design, where every user types the same question. No prompt injection was tested.

Up next in RAG and vector search

More in RAG and vector search

All RAG and vector search videos

Related videos