#141 Python for data 6.5 min
Web Scraping Responsibly in Python: a robots.txt Trap
On Python 3.12, the built-in robots.txt parser and the RFC 9309 longest-match rule differed on 2 of 8 rows.
Slow the crawler down
Copy the code folder first. In robots.py change Crawl-delay: 2 to Crawl-delay: 5 and run python run.py crawl. Predict the planned times in the log before you run it. When I ran it, the four requests were planned at 0 s, 5 s, 10 s and 15 s, and nothing was slept.
A hint
The clock only moves after a real request, not after a skip.
Be the shut-out bot
In crawler.py change AGENT = "LearningITBot" to AGENT = "GreedyBot" and run python run.py crawl. Predict how many requests reach the stand-in site and why. When I ran it, the first path was skipped with 'robots.txt says no' and zero requests reached the stand-in site.
A hint
Look at the GreedyBot group in the made-up robots file.
Reorder two rules
In robots.py, in ROBOTS_TXT, swap the lines Allow: /shop/ and Disallow: /shop/cart/ so the longer Disallow comes first. Predict which row of the table changes in the python column, then run python run.py rules. When I ran it on Python 3.13, the checkout row for LearningITBot changed to no in the python column and the count became 1 of 8 rows differ. Your Python release may differ, which is the video's point.
A hint
Only the python column can change. The longest-match column does not depend on the order.
Add a path to the table
In run.py add the path "/shop/kettle" to the PATHS list, after "/shop/". Predict what both columns will say for /shop/kettle for LearningITBot and for GreedyBot. Then run python run.py rules and compare. When I ran it, LearningITBot got yes in both columns and GreedyBot got no. The text 'of 8 rows' stayed as it was, because the 8 is typed into the code, so you would need to change it too.
A hint
Remember that /shop/ is allowed for everyone except GreedyBot.
These tasks are for you to try on your own machine. Nothing is checked or saved.
What this does not show
A made-up robots.txt and a stand-in site held in memory; no real website is contacted; delays are computed, not slept; my longest-match check has no wildcards; Python's parser was run on 3.12.15 and 3.13.2 only, and newer releases may answer differently.
Up next in Python for data
More in Python for data
- #08 Pandas vs Polars vs DuckDB on 10M Rows, Timed and Checked
- #09 Clean Messy Data with Pandas 3: End to End, 3 Mistakes
- #137 Polars for pandas Users: One Report, Translated
- #138 Seven pandas Changes Tested on 1M Rows: Speed and Memory
- #139 Validate Data with Pydantic: Catch Bad Rows
- #140 Python Dates and Time Zones: 5 Bugs Reproduced
Related videos
- #01 Local AI Agent in Python: No API Key (Ollama) AI agents and MCP
- #02 RAG From Scratch in Python: Chunk, Embed, Retrieve (Local) RAG and vector search
- #03 pgvector vs Chroma vs Qdrant: Same Data, Real Results RAG and vector search
- #05 Structured Outputs in Python: Valid JSON Is Not Correct JSON LLM engineering and evals
