learning-it by example

#48 RAG and vector search 4.9 min

Embedding Models Compared: 6 Local Models, One Test

The smallest embedding model got 16 of 30. Three others got all 30, and one is a third the size.

1. Why does the video say to test embedding models on your own pages rather than rely on a leaderboard alone?
2. In the video's fastembed test on 30 made-up, look-alike shop pages, how many questions did the tiny potion-base-8M embedding model put the right page first for?
3. Why were the 30 test pages made alike on purpose, with the same topics repeated for six products?
4. In the video's fastembed test, bge-small, bge-base and nomic-embed-text-v1.5-Q all got 30 of 30 first places on 30 look-alike shop pages. What does the video conclude about those three?
5. According to the video, which text prefixes do the arctic-embed-xs and nomic-embed-text-v1.5 embedding models expect when used with fastembed?

Practice only: nothing is saved and no score counts towards anything. Each answer comes with the reason.

  1. Look at the test pages

    Copy the code folder first. Open a Python prompt in it and run from shop import build, then pages, questions = build(). Print len(pages), pages[0], questions[0] and pages[29]. Then print every page that mentions helmet. Say in one sentence what makes these pages hard to tell apart.

    A hint

    In my run there were 30 pages and 30 questions, and five pages mention the helmet, but the return page says Helmets with a capital H, so test 'helmet' in page.lower() or you will find only four. A question is a pair of the question text and the number of its right page.

  2. Add a seventh product

    In shop.py add a seventh item to ITEMS, for example ("chain", "chains"). Predict how many pages and questions build() will now return, and what the last page will say. Then, in a fresh Python prompt (an open prompt keeps the old shop module), run it to check. Why does each topic add one page per product? When you have checked, remove the chain item again so shop.py is back to six products, because the later exercises and compare.py assume 30 pages.

    A hint

    There are 5 topics. When I added a chain, the result was 35 pages and the last page was The chain is restocked every 6 weeks.

  3. Write your own question

    Choose one page from build() and write a new question for it in different words from the page, with no copied phrases. Add it to your own list as a pair with the page number. Predict whether a small embedding model would put the right page first, and explain why copying words from the page would make every model look better than it is.

    A hint

    The video lists questions copied from the pages as a mistake.

  4. Predict the ranking

    If fastembed is installed as in the video, run python compare.py and write your prediction first for each of the six models: first place count out of 30. Compare with the video, where the tiny potion model got 16 and all MiniLM got 28. Which models tied, and what would you change to make a harder test? The first run downloads the models, the biggest about 200 megabytes.

    A hint

    Try making the pages closer together, for example more products and the same topics. Your numbers may differ slightly from mine.

These tasks are for you to try on your own machine. Nothing is checked or saved.

What this does not show

Thirty made-up template pages in English with questions I wrote, one run; three models tied at 30, so this test cannot rank them; speed was not measured.

Up next in RAG and vector search

More in RAG and vector search

All RAG and vector search videos

Related videos