#55 RAG and vector search 5 min
Image Search With CLIP in Python: Tested on 12 Drawings
I searched twelve drawn pictures with sentences. It found the blue square. Left of a square, it got wrong.
Read the twelve pictures
Copy the code folder first. Open pictures.py and list, in your own words, the twelve sentences in PICTURES. Group them into the three kinds the video used: colour and shape, how many, and where things are. Then predict which group will score worst when CLIP searches them, and write down your guess before you run anything.
A hint
The video printed one score per group. Your guess comes from what you think a model trained on captions finds hard.
Predict the printed counts
Read compare.py and predict the three score lines it prints and the lines for wrong answers. Then run python compare.py if you have fastembed 0.9.1 installed (the two CLIP models download once, about 600 megabytes). Compare your prediction with the output and note which sentences found the wrong picture.
A hint
The video's own run printed counts of 5 of 6, 2 of 3 and 1 of 3. Your numbers may differ on another machine.
Add a sentence of your own
In pictures.py, add a thirteenth entry to the end of PICTURES: one green square drawn with the BIG box, with the text 'a green square'. In compare.py change range(12) in the wrong list to range(13) so the new sentence is checked. Predict whether 'a green square' will find its own picture, then run it and see.
A hint
Copy the line for the blue square and change the colour and the text. If 'a green square' finds its own picture, nothing extra is printed for it, because only wrong answers are listed.
Test positions you choose
Add two entries to the end of PICTURES: a blue square above a red circle, and a yellow circle left of a green triangle (circle in the LEFT box, triangle in the RIGHT box), each with its own sentence. Do not add a red circle right of a blue square: it draws the same picture as the existing blue square left of a red circle. Use the TOP, LOW, LEFT and RIGHT boxes in pictures.py. In compare.py change range(12) to range(len(PICTURES)). Predict which sentences find a wrong picture, run it, and write one line on whether positions look harder than colours. A few pictures show only a pattern.
A hint
Copy the entry for the red circle above a blue square and swap the shapes and colours. For the second one, copy the entry for the red circle left of a blue square and change the shapes and colours. The three group lines still count only the first twelve sentences, but their numbers can change.
These tasks are for you to try on your own machine. Nothing is checked or saved.
What this does not show
Twelve pictures drawn by the code, plain shapes on white, with sentences I wrote. One CLIP model of one size, one run. Twelve pictures cannot say how it behaves across thousands of real photos: a pattern, not a verdict.
Up next in RAG and vector search
More in RAG and vector search
- #02 RAG From Scratch in Python: Chunk, Embed, Retrieve (Local)
- #03 pgvector vs Chroma vs Qdrant: Same Data, Real Results
- #40 Chunking Strategies Compared on One Document
- #41 Hybrid Search: BM25 Plus Vectors, Tested
- #42 Reranking With a Cross-Encoder: Does It Help? Python Test
- #43 Evaluate RAG: Recall, Faithfulness and a Wrong Checker
Related videos
- #293 RAG for Beginners: The Idea, Then 37 Lines of Python RAG and vector search
- #42 Reranking With a Cross-Encoder: Does It Help? Python Test RAG and vector search
- #48 Embedding Models Compared: 6 Local Models, One Test RAG and vector search
- #44 RAG Over PDFs: Check Your Extracted Text Before You Index RAG and vector search
