Long Context vs RAG: When Can You Skip Retrieval?
Pasting everything into a 4,096 token prompt got four of four right, until the catalogue doubled.
Video #54 ยท RAG and vector search
Steps in the video
Files
- ask.py (28 lines)
- catalogue.py (31 lines)
- compare.py (18 lines)
Tested with
Python 3.12.15 | ollama==0.6.3
What this does not show
Four questions, one small model (llama3.2), and a context I set to 4,096 tokens; a model with a longer context would fit all 600 pages. Every question is about the start of the catalogue, so the middle was not tested. The catalogue is made up, with one fact per line, which makes retrieval easy.
Sources
Learning IT by Example.