Fine-Tuning vs RAG vs Prompting: A Test You Can Run
Prompting, pasted context and retrieval tested on four questions with a local model: plus a simple paste test for whether you need RAG. Fine-tuning is explained, not run.
Video #10 ยท Local and open models
Steps in the video
- 0:00 Which fix should you try?
- 0:20 Three different fixes
- 0:58 Setup
- 1:18 A handbook the model has never seen
- 1:45 Four questions and a scorer
- 2:13 Pasting and retrieving
- 2:40 Three ways to answer
- 3:29 Reading the results
- 4:00 Where fine-tuning fits
- 4:34 What this does not show
- 5:06 Mistakes to avoid
- 5:34 My take and next steps
- 6:06 Recap
Files
- ask.py (56 lines)
- handbook.txt (6 lines)
Tested with
Python 3.12.15 | ollama==0.6.3
What this does not show
Four questions on an invented six-line handbook with one small local model are not a benchmark. Fine-tuning is explained from vendor and practitioner blogs and is not run. The script does not print the retrieved line, so the cause of the one wrong retrieval answer is not shown.
Sources
- https://blog.premai.io/fine-tuning-vs-rag-a-decision-framework-for-custom-llm-applications
- https://dev.to/emperorakashi20/prompting-vs-rag-vs-fine-tuning-when-to-use-each-a-developers-decision-framework-34nd
- https://memx.app/blog/fine-tuning-vs-rag-which-to-use/
- https://pypi.org/project/ollama/0.6.3/
- https://tianpan.co/blog/2026-04-10-finetuning-vs-rag-knowledge-injection
- https://www.azion.com/en/learning/ai/fine-tuning-vs-rag/
- https://www.raftlabs.com/blog/llm-fine-tuning-guide
Learning IT by Example.