{"data":{"id":183,"slug":"respecting-robots-txt-in-practice-part-10-183","title":"Respecting robots.txt in practice, part 10","excerpt":"In part 10 we take the ideas from earlier instalments and apply them to a larger, messier site. robots.txt is a short text file with a long history. It tells automated clients which parts of a site…","seriesName":"Respecting robots.txt in practice","seriesPart":10,"sections":[{"heading":"Introduction","paragraphs":["In part 10 we take the ideas from earlier instalments and apply them to a larger, messier site. robots.txt is a short text file with a long history. It tells automated clients which parts of a site the operator would prefer they left alone, and how quickly they should crawl the rest."]},{"heading":"Putting it into practice","paragraphs":["Read the site's terms of service before you start. Many sites welcome automated access within limits and say so explicitly; others ask you to use an API instead.","Log every request with its URL, status code and duration. When something goes wrong at two in the morning, those logs are the difference between a quick fix and a long investigation.","Pagination comes in several shapes: numbered pages, next links, offsets and opaque cursors. Each has trade-offs, and a robust scraper should detect the end of the result set rather than assuming a fixed number of pages."]},{"heading":"Step by step","paragraphs":["Write small, focused tests for your parsers using saved HTML fixtures. They run in milliseconds and catch layout changes long before a full crawl would.","Be careful with relative URLs. Resolve every link against the page it came from, not against the site root, or you will request pages that do not exist.","Validate the data you extract. Prices without currencies, dates without time zones and empty fields are common sources of silent errors downstream."]},{"heading":"Key takeaway","paragraphs":["Fetch robots.txt before you crawl, honour Disallow rules and Crawl-delay values, and re-check it periodically because sites change their policies."]}],"authorId":4,"authorName":"Lucas Silva","authorSlug":"lucas-silva","category":"Ethics & Law","tags":["scrapy","playwright","ethics"],"publishedAt":"2025-08-03T00:00:00.000Z","wordCount":243,"readingTimeMinutes":2}}