learning-it by example

#46 RAG and vector search 5.3 min

GraphRAG Explained With a Tiny Graph in Python

With two sentences, text search had the facts for 1 of 6 questions. A graph written by hand had them for all 6.

1. Why can ordinary vector search struggle with a question like who manages the person who wrote the Leeds safety report?
2. In the tiny GraphRAG demo, the graph written by hand had all the needed facts for 6 of 6 questions. How many of the answers were right?
3. Why could the top 2 text search never have the facts for all six questions?
4. How does the neighbourhood function decide which facts to hand to the model for a question?
5. The graph built by the model matched only 8 of the 13 true links. What mistake does the video say this warns against?

Practice only: nothing is saved and no score counts towards anything. Each answer comes with the reason.

  1. Count the facts by hops

    Copy the code folder first. Open a Python prompt in it and import neighbourhood and gold_triples from graph. Predict how many facts neighbourhood("Who manages Sam?", gold_triples(), hops=1) returns, then try hops of 1, 2, 3 and 4 and write down the counts. What happens to the context as you follow the links further?

    A hint

    Use len() on the result. When I ran it the counts were 2, 4, 8 and 10 out of 13 facts.

  2. A name that is not in the graph

    Call neighbourhood("Who is Kim?", gold_triples()). Predict what comes back, then run it. Now add one more triple to a copy of the list, ("Kim", "manages", "Ines"), and ask "Who manages the person who runs the Bristol depot?" before and after. Compare the two lists of facts.

    A hint

    The code only starts from names that appear in the question. Make the copy with gold_triples() + [the new triple]. In my run, the new fact Kim manages Ines appeared in the list.

  3. How many hops are enough

    In compare.py the graph search uses three steps. For each of hops 1, 2 and 3, count how many of the six questions in QUESTIONS get all the documents they need from the hand written graph. Use the holds function from compare.py, copied into your own script so that you do not have to run the model. Predict the three counts first.

    A hint

    Copy holds into a new script that imports DOCS and QUESTIONS from corpus and neighbourhood and gold_triples from graph. My run gave 1, 5 and 6 questions.

  4. Predict the answers

    Without running the model, read the six questions in corpus.py and write down, for each one, the answer you expect and the documents it needs. Then run python compare.py only if Ollama is already installed with llama3.2 and all-minilm. Compare your six answers with the video, where the hand written graph had all the facts for 6 of 6 but the model answered only 3 of 6 right. Which questions might the small model get wrong, and why?

    A hint

    Questions that need three facts joined are harder for a small model to reason through, even when all the facts are in front of it.

These tasks are for you to try on your own machine. Nothing is checked or saved.

What this does not show

Thirteen made-up sentences, six questions and one small model in one seeded run; a toy graph, not Microsoft GraphRAG (no communities or summaries); text search was held to two sentences while the hand-written graph handed over five to eight of the thirteen.

Up next in RAG and vector search

More in RAG and vector search

All RAG and vector search videos

Related videos