This is part 9 of our series, and this time we focus on edge cases readers asked about. robots.txt is a short text file with a long history. It tells automated clients which parts of a site the operator would prefer they left alone, and how quickly they should crawl the rest.
Step by step
Read the site's terms of service before you start. Many sites welcome automated access within limits and say so explicitly; others ask you to use an API instead.
Treat HTTP 429 responses as a signal, not an obstacle. Read the Retry-After header, back off, and resume at a slower pace.
Keep configuration such as base URLs, delays and selectors out of your code. Changing a delay should not require a redeploy.
Common pitfalls
Time zones matter more than they seem. Store timestamps in UTC and convert only when displaying them.
Pagination comes in several shapes: numbered pages, next links, offsets and opaque cursors. Each has trade-offs, and a robust scraper should detect the end of the result set rather than assuming a fixed number of pages.
Key takeaway
Fetch robots.txt before you crawl, honour Disallow rules and Crawl-delay values, and re-check it periodically because sites change their policies.