In part 8 we take the ideas from earlier instalments and apply them to a larger, messier site. Before you crawl a site link by link, check whether it already publishes a list of every page it wants found. That list is usually called sitemap.xml.
Putting it into practice
Treat HTTP 429 responses as a signal, not an obstacle. Read the Retry-After header, back off, and resume at a slower pace.
Selectors are the most fragile part of any scraper. Prefer stable attributes such as data-testid or semantic structure over generated class names, which can change on every deploy.
Common pitfalls
Character encodings still cause trouble. Decode responses using the charset the server declares, and normalise text before comparing or storing it.
Time zones matter more than they seem. Store timestamps in UTC and convert only when displaying them.
Key takeaway
Look for the Sitemap line in robots.txt, follow sitemap index files to their children, and use lastmod to skip pages that have not changed.