This is part 4 of our series, and this time we focus on edge cases readers asked about. Scrapy makes it very easy to send thousands of requests a minute. Making a spider that a site operator would be happy to see in their logs takes a few deliberate settings.
Going further
Deduplicate as you go. The same item often appears on several listing pages, and a stable identifier such as a SKU or ID makes duplicates easy to spot.
Keep configuration such as base URLs, delays and selectors out of your code. Changing a delay should not require a redeploy.
A practical approach
Validate the data you extract. Prices without currencies, dates without time zones and empty fields are common sources of silent errors downstream.
Pagination comes in several shapes: numbered pages, next links, offsets and opaque cursors. Each has trade-offs, and a robust scraper should detect the end of the result set rather than assuming a fixed number of pages.
Key takeaway
Turn on ROBOTSTXT_OBEY, set AutoThrottle, use a descriptive User-Agent with contact details, and cache responses while you develop.