learning-it by example

#40 RAG and vector search 5.5 min

Chunking Strategies Compared on One Document

Chunking decides what your RAG app can find. I cut one document five ways and scored each against ten questions.

1. In a chunking comparison on a made-up bike shop handbook, what does the 'whole' check ask, and what does it need?
2. In the chunking comparison on the made-up 586-word handbook with ten questions, using the all-minilm embedding model, how did the five cuts score at the top result?
3. In the made-up bike shop handbook, which answer was cut in half by the fixed 500 character chunks, and where?
4. What does overlap do in the fixed size chunker, and what does it cost?
5. Why does the chunking comparison say its results are a method and not a ranking?

Practice only: nothing is saved and no score counts towards anything. Each answer comes with the reason.

  1. Find the cut answer

    Copy the code folder first. Run python cut.py. It needs no model. Read the output and say which question lost its answer and where the cut fell. I got the voucher question, with the text cut in the middle of the word swapped.

    A hint

    The script prints a little of the text before the cut and after it.

  2. Count chunks and whole answers

    Create try.py in the code folder with: from chunkers import fixed, then from golden import GOLDEN, then text = open('handbook.md', encoding='utf-8').read(). Make chunks = fixed(text, 500) and print len(chunks) and sum(any(n in c for c in chunks) for _, n in GOLDEN). Predict, then run. Repeat with overlap=100. I got 7 chunks and 9 whole answers, then 8 chunks and 10.

    A hint

    The second number counts the phrases that appear whole in at least one chunk.

  3. A smaller overlap

    Run the same count with overlap values 0, 10, 50 and 100 at size 500. Predict first which values reach 10 of 10. I got 9 for 0, and 10 for 10, 50 and 100. Then write one sentence on why even a small overlap rescued this handbook, and why it might not on another document.

    A hint

    The chunks now start every 500 minus the overlap characters, so every later cut moves too. Check whether a cut still falls inside the voucher phrase.

  4. Which question goes missing at 300

    Print the numbers of the questions whose phrase is not whole in any chunk, for sizes 300 and 500 with no overlap. Use enumerate(GOLDEN, 1). Predict, then run. I got question 4 for size 300 and question 8 for size 500. Say what that tells you about choosing a chunk size without measuring.

    A hint

    Question 4 is about the frame warranty.

These tasks are for you to try on your own machine. Nothing is checked or saved.

What this does not show

One made-up 586 word handbook and ten questions I wrote, so one question is a ten percent swing: a method, not a ranking. Clean headings favour the heading split. One small embedding model. No semantic chunking tried.

Up next in RAG and vector search

More in RAG and vector search

All RAG and vector search videos

Related videos