learning-it by example

#10 Local and open models 6.7 min

Fine-Tuning vs RAG vs Prompting: A Test You Can Run

Prompting, pasted context and retrieval tested on four questions with a local model: plus a simple paste test for whether you need RAG. Fine-tuning is explained, not run.

1. The video says the three fixes change three different things. What does fine-tuning change?
2. Pasting the whole handbook into the prompt gave correct answers. What does the video conclude from that?
3. How many of the four questions did the retrieve one paragraph method get right in the fine-tuning, RAG and prompting test?
4. What does the video say about fine-tuning in this test?
5. Why does the video list fine-tuning to add facts as a mistake?

Practice only: nothing is saved and no score counts towards anything. Each answer comes with the reason.

  1. Add a seventh fact and a fifth question

    Copy the code folder first. Add a new line to handbook.txt, such as a Parking line with an invented rule. In ask.py add a fifth entry to QUESTIONS with a question about it and the words a correct answer must contain. Predict the three score lines before you run python ask.py. Does prompting only get your new question right?

    A hint

    The model cannot know an invented fact, which is why the video made the handbook up. The score lines now say 'of 5'.

  2. Change a fact but not the scorer

    In handbook.txt change the refund window from 21 days to 14 days, and leave ask.py as it is. Predict which of the three score lines will change, and why. Run it. Then fix the wanted words in QUESTIONS and run again.

    A hint

    is_right only checks whether certain words appear in the answer. It does not know what is true.

  3. See what retrieval picked

    At the bottom of ask.py add a loop over QUESTIONS that prints each question and the paragraph retrieve returns for it. Predict whether the gift card question retrieves the gift card line, then run it. Does this tell you whether to fix the index or the prompt?

    A hint

    The video says this script does not print the retrieved paragraph, so it cannot say whether retrieval or the small model was at fault.

  4. Retrieve two paragraphs

    Change retrieve so it returns the best two handbook lines, joined with a newline, instead of just one. Predict whether the retrieve one paragraph score and its gift card answer change, then run it. Think about what this costs when your documents are large.

    A hint

    Try sorted(zip(DOCS, VECTORS), key=lambda pair: cosine(q, pair[1]), reverse=True) and join the first two lines.

These tasks are for you to try on your own machine. Nothing is checked or saved.

What this does not show

Four questions on an invented six-line handbook with one small local model are not a benchmark. Fine-tuning is explained from vendor and practitioner blogs and is not run. The script does not print the retrieved line, so the cause of the one wrong retrieval answer is not shown.

Up next in Local and open models

More in Local and open models

All Local and open models videos

Related videos