learning-it by example

#141 Python for data 6.5 min

Web Scraping Responsibly in Python: a robots.txt Trap

On Python 3.12, the built-in robots.txt parser and the RFC 9309 longest-match rule differed on 2 of 8 rows.

1. When two robots.txt rules both match a path, which one does RFC 9309 say a crawler must use, and what wins a tie?
2. On Python 3.12, the built-in robots.txt parser was compared with a longest-match check on a made-up file, using 8 rows of agents and paths. How many rows differed, and how many differed with the longer rules moved first?
3. The made-up crawl ran twice over the stand-in site. What did the two passes show?
4. According to the video, does a robots.txt file work as access control?
5. How does the demo crawler apply the 2 second crawl delay between requests?

Practice only: nothing is saved and no score counts towards anything. Each answer comes with the reason.

  1. Slow the crawler down

    Copy the code folder first. In robots.py change Crawl-delay: 2 to Crawl-delay: 5 and run python run.py crawl. Predict the planned times in the log before you run it. When I ran it, the four requests were planned at 0 s, 5 s, 10 s and 15 s, and nothing was slept.

    A hint

    The clock only moves after a real request, not after a skip.

  2. Be the shut-out bot

    In crawler.py change AGENT = "LearningITBot" to AGENT = "GreedyBot" and run python run.py crawl. Predict how many requests reach the stand-in site and why. When I ran it, the first path was skipped with 'robots.txt says no' and zero requests reached the stand-in site.

    A hint

    Look at the GreedyBot group in the made-up robots file.

  3. Reorder two rules

    In robots.py, in ROBOTS_TXT, swap the lines Allow: /shop/ and Disallow: /shop/cart/ so the longer Disallow comes first. Predict which row of the table changes in the python column, then run python run.py rules. When I ran it on Python 3.13, the checkout row for LearningITBot changed to no in the python column and the count became 1 of 8 rows differ. Your Python release may differ, which is the video's point.

    A hint

    Only the python column can change. The longest-match column does not depend on the order.

  4. Add a path to the table

    In run.py add the path "/shop/kettle" to the PATHS list, after "/shop/". Predict what both columns will say for /shop/kettle for LearningITBot and for GreedyBot. Then run python run.py rules and compare. When I ran it, LearningITBot got yes in both columns and GreedyBot got no. The text 'of 8 rows' stayed as it was, because the 8 is typed into the code, so you would need to change it too.

    A hint

    Remember that /shop/ is allowed for everyone except GreedyBot.

These tasks are for you to try on your own machine. Nothing is checked or saved.

What this does not show

A made-up robots.txt and a stand-in site held in memory; no real website is contacted; delays are computed, not slept; my longest-match check has no wildcards; Python's parser was run on 3.12.15 and 3.13.2 only, and newer releases may answer differently.

Up next in Python for data

More in Python for data

All Python for data videos

Related videos