learning-it by example

Back to the video

Fine-Tuning vs RAG vs Prompting: A Test You Can Run

Prompting, pasted context and retrieval tested on four questions with a local model: plus a simple paste test for whether you need RAG. Fine-tuning is explained, not run.

Video #10 ยท Local and open models

Steps in the video

  1. 0:00 Which fix should you try?
  2. 0:20 Three different fixes
  3. 0:58 Setup
  4. 1:18 A handbook the model has never seen
  5. 1:45 Four questions and a scorer
  6. 2:13 Pasting and retrieving
  7. 2:40 Three ways to answer
  8. 3:29 Reading the results
  9. 4:00 Where fine-tuning fits
  10. 4:34 What this does not show
  11. 5:06 Mistakes to avoid
  12. 5:34 My take and next steps
  13. 6:06 Recap

Files

  • ask.py (56 lines)
  • handbook.txt (6 lines)

Download the code (zip)

Tested with

Python 3.12.15 | ollama==0.6.3

What this does not show

Four questions on an invented six-line handbook with one small local model are not a benchmark. Fine-tuning is explained from vendor and practitioner blogs and is not run. The script does not print the retrieved line, so the cause of the one wrong retrieval answer is not shown.

Sources

Learning IT by Example.