{"data":{"id":64,"slug":"building-a-polite-scrapy-spider-part-4-64","title":"Building a polite Scrapy spider, part 4","excerpt":"This is part 4 of our series, and this time we focus on edge cases readers asked about. Scrapy makes it very easy to send thousands of requests a minute. Making a spider that a site operator would be…","seriesName":"Building a polite Scrapy spider","seriesPart":4,"sections":[{"heading":"Introduction","paragraphs":["This is part 4 of our series, and this time we focus on edge cases readers asked about. Scrapy makes it very easy to send thousands of requests a minute. Making a spider that a site operator would be happy to see in their logs takes a few deliberate settings."]},{"heading":"Going further","paragraphs":["Deduplicate as you go. The same item often appears on several listing pages, and a stable identifier such as a SKU or ID makes duplicates easy to spot.","Keep configuration such as base URLs, delays and selectors out of your code. Changing a delay should not require a redeploy."]},{"heading":"A practical approach","paragraphs":["Validate the data you extract. Prices without currencies, dates without time zones and empty fields are common sources of silent errors downstream.","Pagination comes in several shapes: numbered pages, next links, offsets and opaque cursors. Each has trade-offs, and a robust scraper should detect the end of the result set rather than assuming a fixed number of pages."]},{"heading":"Key takeaway","paragraphs":["Turn on ROBOTSTXT_OBEY, set AutoThrottle, use a descriptive User-Agent with contact details, and cache responses while you develop."]}],"authorId":9,"authorName":"Mateo Hughes","authorSlug":"mateo-hughes","category":"Scraping Basics","tags":["ethics"],"publishedAt":"2024-06-27T00:00:00.000Z","wordCount":175,"readingTimeMinutes":2}}