learning-it by example

Back to the video

Web Scraping Responsibly in Python: a robots.txt Trap

On Python 3.12, the built-in robots.txt parser and the RFC 9309 longest-match rule differed on 2 of 8 rows.

Video #141 ยท Python for data

Steps in the video

  1. 0:00 On Python 3.12, its robots.txt parser and the RFC rule differed on 2 of 8 rows
  2. 0:33 The idea
  3. 0:55 Setup
  4. 1:12 The made-up robots file
  5. 1:31 Python's parser
  6. 1:49 The longest match
  7. 2:24 A controlled test
  8. 3:04 Python is changing
  9. 3:30 Say who you are
  10. 3:49 Robots first
  11. 4:00 Cache and delay
  12. 4:46 What this does not show
  13. 5:20 Mistakes to avoid
  14. 5:49 My take and next steps
  15. 6:07 Recap

Files

  • crawler.py (39 lines)
  • robots.py (55 lines)
  • run.py (46 lines)
  • standin.py (22 lines)

Download the code (zip)

Tested with

Python 3.12.15

What this does not show

A made-up robots.txt and a stand-in site held in memory; no real website is contacted; delays are computed, not slept; my longest-match check has no wildcards; Python's parser was run on 3.12.15 and 3.13.2 only, and newer releases may answer differently.

Sources

Learning IT by Example.