learning-it by example

#44 RAG and vector search 5.4 min

RAG Over PDFs: Check Your Extracted Text Before You Index

One page of my PDF came back with no text of its own, and nothing warned me.

1. Why should you read the text a PDF extractor returns before you build a search on it?
2. In the four page cafe handbook, how many header and footer lines did plain extraction carry into the text before any cleanup?
3. After the repeated lines were dropped, one page had no text left. What was that page, and what does it need?
4. How does the drop_repeats cleanup treat the footers Page 1 and Page 2 as the same line?
5. Why does a clean result on the code generated cafe PDF not promise a clean result on your own PDFs?

Practice only: nothing is saved and no score counts towards anything. Each answer comes with the reason.

  1. Predict the page count

    Copy the code folder first. In make_pdf.py the plain text sections use the string "Plain text for this section. " repeated 12 times. Predict what happens to the header and footer line count printed by compare.py if you change 12 to 200 so that long sections spill onto extra pages. Then run python compare.py and compare with your prediction. When you are done, change 200 back to 12, so the later exercises use the original four page PDF.

    A hint

    Each page carries one header line and one footer line. With the original 4 pages the video showed 8 such lines.

  2. Run the cleanup on your own pages

    Open a Python prompt in the code folder and import drop_repeats from extract.py. Make a list of three strings, each with a first line Cafe - internal, a second line Page 1, Page 2 and Page 3, and then one line of your own text. Predict, then call drop_repeats on the list and print each page. Check that the header and footer lines are gone and your own lines stay.

    A hint

    Join the lines of each string with backslash n. The function returns a list of lines for each page.

  3. Break the repeat rule

    Use the same three pages, but delete the Cafe - internal line from the third page only. Predict what drop_repeats now does with the header on pages one and two, then run it and see. Write one sentence explaining why the function only removes lines that appear on every page.

    A hint

    The function keeps the lines that all pages have in common. When I ran this, the header stayed on pages one and two.

  4. Find the empty page

    After python compare.py has built handbook.pdf, use plain and drop_repeats from extract.py on it. Loop over the pages and print the page number of any page with no text left after the cleanup. Predict the number first, counting pages from 1, then check it and say what that page contains.

    A hint

    Use enumerate(kept, start=1) and test with any(line.strip() for line in page).

These tasks are for you to try on your own machine. Nothing is checked or saved.

What this does not show

The PDF is generated by code, which is the easy case. Real PDFs have two columns, rotated text, odd fonts and merged table cells, and will be messier. No OCR was run, so the scanned page is only detected, not read. One small table is a very small test.

Up next in RAG and vector search

More in RAG and vector search

All RAG and vector search videos

Related videos