learning-it by example

Back to the video

Long Context vs RAG: When Can You Skip Retrieval?

Pasting everything into a 4,096 token prompt got four of four right, until the catalogue doubled.

Video #54 ยท RAG and vector search

Steps in the video

  1. 0:00 Put everything in the prompt, until it stopped working
  2. 0:29 The idea
  3. 1:01 Setup
  4. 1:31 A made up catalogue
  5. 2:03 Two ways to answer
  6. 2:32 The comparison
  7. 3:31 What this does not show
  8. 4:01 Mistakes to avoid
  9. 4:32 My take and next steps
  10. 5:00 Recap

Files

  • ask.py (28 lines)
  • catalogue.py (31 lines)
  • compare.py (18 lines)

Download the code (zip)

Tested with

Python 3.12.15 | ollama==0.6.3

What this does not show

Four questions, one small model (llama3.2), and a context I set to 4,096 tokens; a model with a longer context would fit all 600 pages. Every question is about the start of the catalogue, so the middle was not tested. The catalogue is made up, with one fact per line, which makes retrieval easy.

Sources

Learning IT by Example.