learning-it by example

#41 RAG and vector search 5.7 min

Hybrid Search: BM25 Plus Vectors, Tested

Keyword search and vector search fail in opposite ways. I merged them and scored all three on twelve questions.

1. In hybrid search (BM25 keyword search plus vector search), what does each find well, and how do they fail?
2. In the hybrid search demo, why are the raw BM25 and cosine scores not simply added together, and what does reciprocal rank fusion use instead?
3. In the hybrid search demo (24 made-up help desk articles, 12 questions, all-minilm embeddings), how many questions did each method put in first place?
4. In the hybrid search demo, why did the question 'when is the invoice emailed' go wrong in the merged (rank fusion) results?
5. What does the video say the results table shows, given 24 articles and 12 questions written by the author?

Practice only: nothing is saved and no score counts towards anything. Each answer comes with the reason.

  1. Fix the plural by hand

    Copy the code folder first. This needs no model. In a new file try.py in the code folder, import BM25 from bm25, DOCS from corpus and ranking from hybrid. Build index = BM25(list(DOCS.values())) and ids = list(DOCS). Print the position of d09 in ranking(index.scores(q)) mapped through ids, for q = 'when is the invoice emailed' and for q = 'when are invoices emailed'. Predict first. I got 17 and 1.

    A hint

    Position is the list index of d09 in the ranked ids, plus 1. The tokenizer does no stemming.

  2. See what the tokenizer does

    In Python, run from bm25 import tokens and print tokens('ERR-4012: The sync agent'). Predict the list first, then compare with what I got: err, 4012, the, sync, agent. Say why a search for ERR-4012 can still find the right article.

    A hint

    The tokenizer lowercases the text and splits on anything that is not a letter or a digit.

  3. Merge two rankings

    From hybrid import rrf and weighted. Predict rrf([[0, 1, 2], [2, 1, 0]]), then run it. I got [0, 2, 1]. Then predict weighted([[5, 5, 5], [1, 2, 3]], [0.5, 0.5]), where the first list is all equal, and run it. I got [2, 1, 0]. Explain why the first list does not break the scaling.

    A hint

    In weighted, the code divides by (high minus low) or 1.

  4. Change k in rank fusion

    Call rrf([[0, 1, 2], [0, 2, 1]]) with the default k, and again with k=1. Predict which document comes first, and work out by hand the gap between the sums for first and second place each time. Then run both calls. Say what a smaller k does to that gap, using the formula 1 over k plus rank.

    A hint

    Compare the sums for documents 0 and 1 by hand before running.

These tasks are for you to try on your own machine. Nothing is checked or saved.

What this does not show

24 made-up articles and 12 questions I wrote, in three kinds on purpose, so the table shows what each method is good at, not how often it wins. One question is 8 points. No stemming, no reranker, k not tuned. One small embedding model.

Up next in RAG and vector search

More in RAG and vector search

All RAG and vector search videos

Related videos