Web Scraping Responsibly in Python: a robots.txt Trap
On Python 3.12, the built-in robots.txt parser and the RFC 9309 longest-match rule differed on 2 of 8 rows.
Video #141 ยท Python for data
Steps in the video
- 0:00 On Python 3.12, its robots.txt parser and the RFC rule differed on 2 of 8 rows
- 0:33 The idea
- 0:55 Setup
- 1:12 The made-up robots file
- 1:31 Python's parser
- 1:49 The longest match
- 2:24 A controlled test
- 3:04 Python is changing
- 3:30 Say who you are
- 3:49 Robots first
- 4:00 Cache and delay
- 4:46 What this does not show
- 5:20 Mistakes to avoid
- 5:49 My take and next steps
- 6:07 Recap
Files
- crawler.py (39 lines)
- robots.py (55 lines)
- run.py (46 lines)
- standin.py (22 lines)
Tested with
Python 3.12.15
What this does not show
A made-up robots.txt and a stand-in site held in memory; no real website is contacted; delays are computed, not slept; my longest-match check has no wildcards; Python's parser was run on 3.12.15 and 3.13.2 only, and newer releases may answer differently.
Sources
Learning IT by Example.