#44 RAG and vector search 5.4 min
RAG Over PDFs: Check Your Extracted Text Before You Index
One page of my PDF came back with no text of its own, and nothing warned me.
Predict the page count
Copy the code folder first. In make_pdf.py the plain text sections use the string "Plain text for this section. " repeated 12 times. Predict what happens to the header and footer line count printed by compare.py if you change 12 to 200 so that long sections spill onto extra pages. Then run python compare.py and compare with your prediction. When you are done, change 200 back to 12, so the later exercises use the original four page PDF.
A hint
Each page carries one header line and one footer line. With the original 4 pages the video showed 8 such lines.
Run the cleanup on your own pages
Open a Python prompt in the code folder and import drop_repeats from extract.py. Make a list of three strings, each with a first line Cafe - internal, a second line Page 1, Page 2 and Page 3, and then one line of your own text. Predict, then call drop_repeats on the list and print each page. Check that the header and footer lines are gone and your own lines stay.
A hint
Join the lines of each string with backslash n. The function returns a list of lines for each page.
Break the repeat rule
Use the same three pages, but delete the Cafe - internal line from the third page only. Predict what drop_repeats now does with the header on pages one and two, then run it and see. Write one sentence explaining why the function only removes lines that appear on every page.
A hint
The function keeps the lines that all pages have in common. When I ran this, the header stayed on pages one and two.
Find the empty page
After python compare.py has built handbook.pdf, use plain and drop_repeats from extract.py on it. Loop over the pages and print the page number of any page with no text left after the cleanup. Predict the number first, counting pages from 1, then check it and say what that page contains.
A hint
Use enumerate(kept, start=1) and test with any(line.strip() for line in page).
These tasks are for you to try on your own machine. Nothing is checked or saved.
What this does not show
The PDF is generated by code, which is the easy case. Real PDFs have two columns, rotated text, odd fonts and merged table cells, and will be messier. No OCR was run, so the scanned page is only detected, not read. One small table is a very small test.
Up next in RAG and vector search
More in RAG and vector search
- #02 RAG From Scratch in Python: Chunk, Embed, Retrieve (Local)
- #03 pgvector vs Chroma vs Qdrant: Same Data, Real Results
- #40 Chunking Strategies Compared on One Document
- #41 Hybrid Search: BM25 Plus Vectors, Tested
- #42 Reranking With a Cross-Encoder: Does It Help? Python Test
- #43 Evaluate RAG: Recall, Faithfulness and a Wrong Checker
Related videos
- #46 GraphRAG Explained With a Tiny Graph in Python RAG and vector search
- #54 Long Context vs RAG: When Can You Skip Retrieval? RAG and vector search
- #55 Image Search With CLIP in Python: Tested on 12 Drawings RAG and vector search
- #05 Structured Outputs in Python: Valid JSON Is Not Correct JSON LLM engineering and evals
