learning-it by example

#55 RAG and vector search 5 min

Image Search With CLIP in Python: Tested on 12 Drawings

I searched twelve drawn pictures with sentences. It found the blue square. Left of a square, it got wrong.

1. What lets CLIP find a picture from a plain sentence?
2. In the video's fastembed test of CLIP (Qdrant/clip-ViT-B-32) on twelve drawn pictures, for how many of the 6 colour and shape sentences did the right picture come first?
3. In the video's CLIP test on twelve drawn pictures, what picture did the two 'left of' sentences find instead of the right one?
4. Why is the CLIP result on twelve drawings a pattern and not a verdict?
5. In the video's CLIP image search, how does the code choose the answer picture for each sentence?

Practice only: nothing is saved and no score counts towards anything. Each answer comes with the reason.

  1. Read the twelve pictures

    Copy the code folder first. Open pictures.py and list, in your own words, the twelve sentences in PICTURES. Group them into the three kinds the video used: colour and shape, how many, and where things are. Then predict which group will score worst when CLIP searches them, and write down your guess before you run anything.

    A hint

    The video printed one score per group. Your guess comes from what you think a model trained on captions finds hard.

  2. Predict the printed counts

    Read compare.py and predict the three score lines it prints and the lines for wrong answers. Then run python compare.py if you have fastembed 0.9.1 installed (the two CLIP models download once, about 600 megabytes). Compare your prediction with the output and note which sentences found the wrong picture.

    A hint

    The video's own run printed counts of 5 of 6, 2 of 3 and 1 of 3. Your numbers may differ on another machine.

  3. Add a sentence of your own

    In pictures.py, add a thirteenth entry to the end of PICTURES: one green square drawn with the BIG box, with the text 'a green square'. In compare.py change range(12) in the wrong list to range(13) so the new sentence is checked. Predict whether 'a green square' will find its own picture, then run it and see.

    A hint

    Copy the line for the blue square and change the colour and the text. If 'a green square' finds its own picture, nothing extra is printed for it, because only wrong answers are listed.

  4. Test positions you choose

    Add two entries to the end of PICTURES: a blue square above a red circle, and a yellow circle left of a green triangle (circle in the LEFT box, triangle in the RIGHT box), each with its own sentence. Do not add a red circle right of a blue square: it draws the same picture as the existing blue square left of a red circle. Use the TOP, LOW, LEFT and RIGHT boxes in pictures.py. In compare.py change range(12) to range(len(PICTURES)). Predict which sentences find a wrong picture, run it, and write one line on whether positions look harder than colours. A few pictures show only a pattern.

    A hint

    Copy the entry for the red circle above a blue square and swap the shapes and colours. For the second one, copy the entry for the red circle left of a blue square and change the shapes and colours. The three group lines still count only the first twelve sentences, but their numbers can change.

These tasks are for you to try on your own machine. Nothing is checked or saved.

What this does not show

Twelve pictures drawn by the code, plain shapes on white, with sentences I wrote. One CLIP model of one size, one run. Twelve pictures cannot say how it behaves across thousands of real photos: a pattern, not a verdict.

Up next in RAG and vector search

More in RAG and vector search

All RAG and vector search videos

Related videos