learning-it by example

#42 RAG and vector search 4.9 min

Reranking With a Cross-Encoder: Does It Help? Python Test

My search found the right page but did not put it first. Can a reranker fix that?

1. What is the two-stage pattern for search with a cross-encoder reranker?
2. In the bike shop help page test (12 made-up pages, 20 questions, one small cross-encoder reranker), how did the reranker change the number of questions with the right page first behind the weak word-overlap search?
3. In the same bike shop help page test, what did the cross-encoder reranker do to the first places behind the embedding (meaning) search?
4. Why could the reranker not rescue 4 of the 20 questions for the word search?
5. In the comparison code, what does the 'made worse' count measure?

Practice only: nothing is saved and no score counts towards anything. Each answer comes with the reason.

  1. Count shared words by hand

    Copy the code folder first. Look at the STOP set and the words function in search.py. For the question 'Is cash accepted?' list the words that remain after the small words are dropped, then list the words of the payment page (index 10 in docs.py), and count how many they share. This is a reading task, because running search.py downloads models.

    A hint

    The word accepted and the word accept are different words to this search.

  2. Shrink the short list

    In search.py change n=5 to n=3 in by_words. Predict whether the number of questions where the top list holds the right page can go up, stay the same or go down compared with the video's 16 of 20, and say why. Run compare.py if you have fastembed installed as in the video. The printed label will still say 'top 5'. Change n back to 5 before the next task.

    A hint

    A list of 3 pages is part of the list of 5 pages.

  3. Rerank everything

    Change n=5 to n=12 in by_words, so the short list is all 12 pages. Predict how many questions will have the right page in the short list, and say why this defeats the point of a first stage. Use what the video says about running the slow model. The printed label will still say 'top 5'.

    A hint

    The video says reranking everything makes the slow model run on the whole collection.

  4. Write a question and predict

    In docs.py add a new question to QUESTIONS, for example 'Can I cancel before it is packed?' with the right page index 11. Work out by hand which words it shares with each page, then predict whether the right page is in the top 5 for the word search. Say what the video would recommend if it is not.

    A hint

    The video's tip is to fix the first stage first if the right page is not in the short list.

These tasks are for you to try on your own machine. Nothing is checked or saved.

What this does not show

Twelve made-up pages and twenty questions that I wrote, one small reranker, one small embedding model, one run on a laptop. A reranker can only reorder what the first stage found. A larger collection may give different numbers.

Up next in RAG and vector search

More in RAG and vector search

All RAG and vector search videos

Related videos