#42 RAG and vector search 4.9 min
Reranking With a Cross-Encoder: Does It Help? Python Test
My search found the right page but did not put it first. Can a reranker fix that?
Count shared words by hand
Copy the code folder first. Look at the STOP set and the words function in search.py. For the question 'Is cash accepted?' list the words that remain after the small words are dropped, then list the words of the payment page (index 10 in docs.py), and count how many they share. This is a reading task, because running search.py downloads models.
A hint
The word accepted and the word accept are different words to this search.
Shrink the short list
In search.py change n=5 to n=3 in by_words. Predict whether the number of questions where the top list holds the right page can go up, stay the same or go down compared with the video's 16 of 20, and say why. Run compare.py if you have fastembed installed as in the video. The printed label will still say 'top 5'. Change n back to 5 before the next task.
A hint
A list of 3 pages is part of the list of 5 pages.
Rerank everything
Change n=5 to n=12 in by_words, so the short list is all 12 pages. Predict how many questions will have the right page in the short list, and say why this defeats the point of a first stage. Use what the video says about running the slow model. The printed label will still say 'top 5'.
A hint
The video says reranking everything makes the slow model run on the whole collection.
Write a question and predict
In docs.py add a new question to QUESTIONS, for example 'Can I cancel before it is packed?' with the right page index 11. Work out by hand which words it shares with each page, then predict whether the right page is in the top 5 for the word search. Say what the video would recommend if it is not.
A hint
The video's tip is to fix the first stage first if the right page is not in the short list.
These tasks are for you to try on your own machine. Nothing is checked or saved.
What this does not show
Twelve made-up pages and twenty questions that I wrote, one small reranker, one small embedding model, one run on a laptop. A reranker can only reorder what the first stage found. A larger collection may give different numbers.
Up next in RAG and vector search
More in RAG and vector search
Related videos
- #48 Embedding Models Compared: 6 Local Models, One Test RAG and vector search
- #55 Image Search With CLIP in Python: Tested on 12 Drawings RAG and vector search
- #293 RAG for Beginners: The Idea, Then 37 Lines of Python RAG and vector search
- #46 GraphRAG Explained With a Tiny Graph in Python RAG and vector search
