#40 RAG and vector search 5.5 min
Chunking Strategies Compared on One Document
Chunking decides what your RAG app can find. I cut one document five ways and scored each against ten questions.
Find the cut answer
Copy the code folder first. Run python cut.py. It needs no model. Read the output and say which question lost its answer and where the cut fell. I got the voucher question, with the text cut in the middle of the word swapped.
A hint
The script prints a little of the text before the cut and after it.
Count chunks and whole answers
Create try.py in the code folder with: from chunkers import fixed, then from golden import GOLDEN, then text = open('handbook.md', encoding='utf-8').read(). Make chunks = fixed(text, 500) and print len(chunks) and sum(any(n in c for c in chunks) for _, n in GOLDEN). Predict, then run. Repeat with overlap=100. I got 7 chunks and 9 whole answers, then 8 chunks and 10.
A hint
The second number counts the phrases that appear whole in at least one chunk.
A smaller overlap
Run the same count with overlap values 0, 10, 50 and 100 at size 500. Predict first which values reach 10 of 10. I got 9 for 0, and 10 for 10, 50 and 100. Then write one sentence on why even a small overlap rescued this handbook, and why it might not on another document.
A hint
The chunks now start every 500 minus the overlap characters, so every later cut moves too. Check whether a cut still falls inside the voucher phrase.
Which question goes missing at 300
Print the numbers of the questions whose phrase is not whole in any chunk, for sizes 300 and 500 with no overlap. Use enumerate(GOLDEN, 1). Predict, then run. I got question 4 for size 300 and question 8 for size 500. Say what that tells you about choosing a chunk size without measuring.
A hint
Question 4 is about the frame warranty.
These tasks are for you to try on your own machine. Nothing is checked or saved.
What this does not show
One made-up 586 word handbook and ten questions I wrote, so one question is a ten percent swing: a method, not a ranking. Clean headings favour the heading split. One small embedding model. No semantic chunking tried.
Up next in RAG and vector search
More in RAG and vector search
Related videos
- #02 RAG From Scratch in Python: Chunk, Embed, Retrieve (Local) RAG and vector search
- #41 Hybrid Search: BM25 Plus Vectors, Tested RAG and vector search
- #68 Synthetic Test Cases: 27 Asked, 17 Passed Checks LLM engineering and evals
- #03 pgvector vs Chroma vs Qdrant: Same Data, Real Results RAG and vector search
