#10 Local and open models 6.7 min
Fine-Tuning vs RAG vs Prompting: A Test You Can Run
Prompting, pasted context and retrieval tested on four questions with a local model: plus a simple paste test for whether you need RAG. Fine-tuning is explained, not run.
Add a seventh fact and a fifth question
Copy the code folder first. Add a new line to handbook.txt, such as a Parking line with an invented rule. In ask.py add a fifth entry to QUESTIONS with a question about it and the words a correct answer must contain. Predict the three score lines before you run python ask.py. Does prompting only get your new question right?
A hint
The model cannot know an invented fact, which is why the video made the handbook up. The score lines now say 'of 5'.
Change a fact but not the scorer
In handbook.txt change the refund window from 21 days to 14 days, and leave ask.py as it is. Predict which of the three score lines will change, and why. Run it. Then fix the wanted words in QUESTIONS and run again.
A hint
is_right only checks whether certain words appear in the answer. It does not know what is true.
See what retrieval picked
At the bottom of ask.py add a loop over QUESTIONS that prints each question and the paragraph retrieve returns for it. Predict whether the gift card question retrieves the gift card line, then run it. Does this tell you whether to fix the index or the prompt?
A hint
The video says this script does not print the retrieved paragraph, so it cannot say whether retrieval or the small model was at fault.
Retrieve two paragraphs
Change retrieve so it returns the best two handbook lines, joined with a newline, instead of just one. Predict whether the retrieve one paragraph score and its gift card answer change, then run it. Think about what this costs when your documents are large.
A hint
Try sorted(zip(DOCS, VECTORS), key=lambda pair: cosine(q, pair[1]), reverse=True) and join the first two lines.
These tasks are for you to try on your own machine. Nothing is checked or saved.
What this does not show
Four questions on an invented six-line handbook with one small local model are not a benchmark. Fine-tuning is explained from vendor and practitioner blogs and is not run. The script does not print the retrieved line, so the cause of the one wrong retrieval answer is not shown.
Up next in Local and open models
More in Local and open models
Related videos
- #01 Local AI Agent in Python: No API Key (Ollama) AI agents and MCP
- #06 Test Your LLM App with pytest: An Eval Harness in 40 Lines LLM engineering and evals
- #12 Multi-Agent Patterns in Plain Python: Pipeline, Router, Loop AI agents and MCP
- #56 Prompt Versioning: Compare Changes Case by Case LLM engineering and evals
