In part 5 we take the ideas from earlier instalments and apply them to a larger, messier site. If a site offers an API for the data you need, scraping its HTML is usually the harder, more fragile and less polite option.
Common pitfalls
Treat HTTP 429 responses as a signal, not an obstacle. Read the Retry-After header, back off, and resume at a slower pace.
Character encodings still cause trouble. Decode responses using the charset the server declares, and normalise text before comparing or storing it.
A practical approach
Be careful with relative URLs. Resolve every link against the page it came from, not against the site root, or you will request pages that do not exist.
Log every request with its URL, status code and duration. When something goes wrong at two in the morning, those logs are the difference between a quick fix and a long investigation.
Key takeaway
Check for a documented API, a data export or a feed first. Scrape only when none exists, and read the terms of service either way.