#48 RAG and vector search 4.9 min
Embedding Models Compared: 6 Local Models, One Test
The smallest embedding model got 16 of 30. Three others got all 30, and one is a third the size.
Look at the test pages
Copy the code folder first. Open a Python prompt in it and run from shop import build, then pages, questions = build(). Print len(pages), pages[0], questions[0] and pages[29]. Then print every page that mentions helmet. Say in one sentence what makes these pages hard to tell apart.
A hint
In my run there were 30 pages and 30 questions, and five pages mention the helmet, but the return page says Helmets with a capital H, so test 'helmet' in page.lower() or you will find only four. A question is a pair of the question text and the number of its right page.
Add a seventh product
In shop.py add a seventh item to ITEMS, for example ("chain", "chains"). Predict how many pages and questions build() will now return, and what the last page will say. Then, in a fresh Python prompt (an open prompt keeps the old shop module), run it to check. Why does each topic add one page per product? When you have checked, remove the chain item again so shop.py is back to six products, because the later exercises and compare.py assume 30 pages.
A hint
There are 5 topics. When I added a chain, the result was 35 pages and the last page was The chain is restocked every 6 weeks.
Write your own question
Choose one page from build() and write a new question for it in different words from the page, with no copied phrases. Add it to your own list as a pair with the page number. Predict whether a small embedding model would put the right page first, and explain why copying words from the page would make every model look better than it is.
A hint
The video lists questions copied from the pages as a mistake.
Predict the ranking
If fastembed is installed as in the video, run python compare.py and write your prediction first for each of the six models: first place count out of 30. Compare with the video, where the tiny potion model got 16 and all MiniLM got 28. Which models tied, and what would you change to make a harder test? The first run downloads the models, the biggest about 200 megabytes.
A hint
Try making the pages closer together, for example more products and the same topics. Your numbers may differ slightly from mine.
These tasks are for you to try on your own machine. Nothing is checked or saved.
What this does not show
Thirty made-up template pages in English with questions I wrote, one run; three models tied at 30, so this test cannot rank them; speed was not measured.
Up next in RAG and vector search
More in RAG and vector search
- #02 RAG From Scratch in Python: Chunk, Embed, Retrieve (Local)
- #03 pgvector vs Chroma vs Qdrant: Same Data, Real Results
- #40 Chunking Strategies Compared on One Document
- #41 Hybrid Search: BM25 Plus Vectors, Tested
- #42 Reranking With a Cross-Encoder: Does It Help? Python Test
- #43 Evaluate RAG: Recall, Faithfulness and a Wrong Checker
Related videos
- #42 Reranking With a Cross-Encoder: Does It Help? Python Test RAG and vector search
- #55 Image Search With CLIP in Python: Tested on 12 Drawings RAG and vector search
- #293 RAG for Beginners: The Idea, Then 37 Lines of Python RAG and vector search
- #49 HNSW vs IVF vs Flat: Vector Indexes Tested on 100k Vectors RAG and vector search
