#54 RAG and vector search 5.4 min
Long Context vs RAG: When Can You Skip Retrieval?
Pasting everything into a 4,096 token prompt got four of four right, until the catalogue doubled.
Look at the catalogue
Copy the code folder first. Open a Python prompt in it and import PRODUCTS, pages and ASKED from catalogue. Predict len(PRODUCTS), len(pages(60)) and len(pages(120)), then print them. Print pages(1) and ASKED and check that every number in the pages differs from the others.
A hint
There are 12 colours and 10 things, and 5 facts per product. When I ran it the lengths were 120, 300 and 600.
Work out a number by hand
The number for a page comes from the function number in catalogue.py. By hand, predict the number on the page The red helmet ships from bay, which is product 0 and fact 4. Then call number("red helmet", 4) to check. Now predict the number for the gold stand, the last product, for the same fact, and call question("gold stand", 4) to see its answer.
A hint
The formula is 100 plus (the product index times 7 plus the fact times 13) modulo 890. When I ran it the red helmet answer was 152 and the gold stand answer was 985.
Retrieve three pages
With Ollama and all-minilm running, build a smaller catalogue with pages(30), embed it with embed from ask.py, and call closest on the question Which bay does the green helmet ship from?. Predict which page comes first, then run it. Does the right page come first, and what are the other two pages?
A hint
closest takes the question, the pages and the vectors. When I ran it, the first page was The green helmet ships from bay 166, followed by the brown and pink helmets.
Count the context
Predict what happens to the whole catalogue answer if you change num_ctx in ask.py from 4096 to a much larger number, for example 16384, given that 600 pages used up the 4096 token limit. If you have the llama3.2 model and enough memory, run python compare.py and compare with the video, where 600 pages gave 0 of 4 with the whole catalogue. Write down why the closest three result at 70 tokens does not depend on the limit. This run can take many minutes.
A hint
The video says a model with a longer context would fit the six hundred pages and the result would change. Your numbers will depend on your machine, so treat them as your own test.
These tasks are for you to try on your own machine. Nothing is checked or saved.
What this does not show
Four questions, one small model (llama3.2), and a context I set to 4,096 tokens; a model with a longer context would fit all 600 pages. Every question is about the start of the catalogue, so the middle was not tested. The catalogue is made up, with one fact per line, which makes retrieval easy.
Up next in RAG and vector search
More in RAG and vector search
- #02 RAG From Scratch in Python: Chunk, Embed, Retrieve (Local)
- #03 pgvector vs Chroma vs Qdrant: Same Data, Real Results
- #40 Chunking Strategies Compared on One Document
- #41 Hybrid Search: BM25 Plus Vectors, Tested
- #42 Reranking With a Cross-Encoder: Does It Help? Python Test
- #43 Evaluate RAG: Recall, Faithfulness and a Wrong Checker
Related videos
- #46 GraphRAG Explained With a Tiny Graph in Python RAG and vector search
- #39 Context Engineering: What to Keep From Chat History AI agents and MCP
- #61 Count Tokens Before You Send: Context Limits LLM engineering and evals
- #69 Detect LLM Hallucinations: 5 Checks Scored on 16 Answers LLM engineering and evals
