<?xml version="1.0" encoding="UTF-8"?>
<response><data><post><id>164</id><slug>building-a-polite-scrapy-spider-part-9-164</slug><title>Building a polite Scrapy spider, part 9</title><excerpt>In part 9 we take the ideas from earlier instalments and apply them to a larger, messier site. Scrapy makes it very easy to send thousands of requests a minute. Making a spider that a site operator…</excerpt><seriesName>Building a polite Scrapy spider</seriesName><seriesPart>9</seriesPart><sections><item><heading>Introduction</heading><paragraphs><item>In part 9 we take the ideas from earlier instalments and apply them to a larger, messier site. Scrapy makes it very easy to send thousands of requests a minute. Making a spider that a site operator would be happy to see in their logs takes a few deliberate settings.</item></paragraphs></item><item><heading>Common pitfalls</heading><paragraphs><item>Be careful with relative URLs. Resolve every link against the page it came from, not against the site root, or you will request pages that do not exist.</item><item>Store raw responses alongside parsed output while you are developing. When a parser breaks, you can replay the saved HTML instead of hitting the site again.</item></paragraphs></item><item><heading>What to watch for</heading><paragraphs><item>Dynamic pages render content after the initial HTML arrives. A browser automation tool can wait for a specific element to appear, which is far more reliable than sleeping for a fixed amount of time.</item><item>Time zones matter more than they seem. Store timestamps in UTC and convert only when displaying them.</item></paragraphs></item><item><heading>Key takeaway</heading><paragraphs><item>Turn on ROBOTSTXT_OBEY, set AutoThrottle, use a descriptive User-Agent with contact details, and cache responses while you develop.</item></paragraphs></item></sections><authorId>8</authorId><authorName>Hannah Novak</authorName><authorSlug>hannah-novak</authorSlug><category>Scraping Basics</category><tags><item>json</item><item>http</item><item>playwright</item><item>scrapy</item></tags><publishedAt>2026-02-04T00:00:00.000Z</publishedAt><wordCount>173</wordCount><readingTimeMinutes>2</readingTimeMinutes></post></data></response>
