In part 9 we take the ideas from earlier instalments and apply them to a larger, messier site. Scrapy makes it very easy to send thousands of requests a minute. Making a spider that a site operator would be happy to see in their logs takes a few deliberate settings.

Common pitfalls

Be careful with relative URLs. Resolve every link against the page it came from, not against the site root, or you will request pages that do not exist.

Store raw responses alongside parsed output while you are developing. When a parser breaks, you can replay the saved HTML instead of hitting the site again.

What to watch for

Dynamic pages render content after the initial HTML arrives. A browser automation tool can wait for a specific element to appear, which is far more reliable than sleeping for a fixed amount of time.

Time zones matter more than they seem. Store timestamps in UTC and convert only when displaying them.

Key takeaway

Turn on ROBOTSTXT_OBEY, set AutoThrottle, use a descriptive User-Agent with contact details, and cache responses while you develop.