learning-it by example

#54 RAG and vector search 5.4 min

Long Context vs RAG: When Can You Skip Retrieval?

Pasting everything into a 4,096 token prompt got four of four right, until the catalogue doubled.

1. What are the two ways the video compares for answering questions about a product catalogue?
2. In the video's Ollama test with llama3.2 (context set to 4,096 tokens, a made-up catalogue of 600 pages), how many of the four questions did the whole catalogue answer right?
3. In the video's Ollama test with llama3.2 on a made-up catalogue of 300 or 600 pages, about how many prompt tokens did the closest three pages use for each question, and how many answers were right?
4. What does the video say happens when a prompt is larger than the context limit, and why is that a mistake to avoid?
5. What does the video say it did not show, because every question was about products near the start of the catalogue?

Practice only: nothing is saved and no score counts towards anything. Each answer comes with the reason.

  1. Look at the catalogue

    Copy the code folder first. Open a Python prompt in it and import PRODUCTS, pages and ASKED from catalogue. Predict len(PRODUCTS), len(pages(60)) and len(pages(120)), then print them. Print pages(1) and ASKED and check that every number in the pages differs from the others.

    A hint

    There are 12 colours and 10 things, and 5 facts per product. When I ran it the lengths were 120, 300 and 600.

  2. Work out a number by hand

    The number for a page comes from the function number in catalogue.py. By hand, predict the number on the page The red helmet ships from bay, which is product 0 and fact 4. Then call number("red helmet", 4) to check. Now predict the number for the gold stand, the last product, for the same fact, and call question("gold stand", 4) to see its answer.

    A hint

    The formula is 100 plus (the product index times 7 plus the fact times 13) modulo 890. When I ran it the red helmet answer was 152 and the gold stand answer was 985.

  3. Retrieve three pages

    With Ollama and all-minilm running, build a smaller catalogue with pages(30), embed it with embed from ask.py, and call closest on the question Which bay does the green helmet ship from?. Predict which page comes first, then run it. Does the right page come first, and what are the other two pages?

    A hint

    closest takes the question, the pages and the vectors. When I ran it, the first page was The green helmet ships from bay 166, followed by the brown and pink helmets.

  4. Count the context

    Predict what happens to the whole catalogue answer if you change num_ctx in ask.py from 4096 to a much larger number, for example 16384, given that 600 pages used up the 4096 token limit. If you have the llama3.2 model and enough memory, run python compare.py and compare with the video, where 600 pages gave 0 of 4 with the whole catalogue. Write down why the closest three result at 70 tokens does not depend on the limit. This run can take many minutes.

    A hint

    The video says a model with a longer context would fit the six hundred pages and the result would change. Your numbers will depend on your machine, so treat them as your own test.

These tasks are for you to try on your own machine. Nothing is checked or saved.

What this does not show

Four questions, one small model (llama3.2), and a context I set to 4,096 tokens; a model with a longer context would fit all 600 pages. Every question is about the start of the catalogue, so the middle was not tested. The catalogue is made up, with one fact per line, which makes retrieval easy.

Up next in RAG and vector search

More in RAG and vector search

All RAG and vector search videos

Related videos