#41 RAG and vector search 5.7 min
Hybrid Search: BM25 Plus Vectors, Tested
Keyword search and vector search fail in opposite ways. I merged them and scored all three on twelve questions.
Fix the plural by hand
Copy the code folder first. This needs no model. In a new file try.py in the code folder, import BM25 from bm25, DOCS from corpus and ranking from hybrid. Build index = BM25(list(DOCS.values())) and ids = list(DOCS). Print the position of d09 in ranking(index.scores(q)) mapped through ids, for q = 'when is the invoice emailed' and for q = 'when are invoices emailed'. Predict first. I got 17 and 1.
A hint
Position is the list index of d09 in the ranked ids, plus 1. The tokenizer does no stemming.
See what the tokenizer does
In Python, run from bm25 import tokens and print tokens('ERR-4012: The sync agent'). Predict the list first, then compare with what I got: err, 4012, the, sync, agent. Say why a search for ERR-4012 can still find the right article.
A hint
The tokenizer lowercases the text and splits on anything that is not a letter or a digit.
Merge two rankings
From hybrid import rrf and weighted. Predict rrf([[0, 1, 2], [2, 1, 0]]), then run it. I got [0, 2, 1]. Then predict weighted([[5, 5, 5], [1, 2, 3]], [0.5, 0.5]), where the first list is all equal, and run it. I got [2, 1, 0]. Explain why the first list does not break the scaling.
A hint
In weighted, the code divides by (high minus low) or 1.
Change k in rank fusion
Call rrf([[0, 1, 2], [0, 2, 1]]) with the default k, and again with k=1. Predict which document comes first, and work out by hand the gap between the sums for first and second place each time. Then run both calls. Say what a smaller k does to that gap, using the formula 1 over k plus rank.
A hint
Compare the sums for documents 0 and 1 by hand before running.
These tasks are for you to try on your own machine. Nothing is checked or saved.
What this does not show
24 made-up articles and 12 questions I wrote, in three kinds on purpose, so the table shows what each method is good at, not how often it wins. One question is 8 points. No stemming, no reranker, k not tuned. One small embedding model.
