This is part 7 of our series, and this time we focus on edge cases readers asked about. Before you crawl a site link by link, check whether it already publishes a list of every page it wants found. That list is usually called sitemap.xml.
What to watch for
Write small, focused tests for your parsers using saved HTML fixtures. They run in milliseconds and catch layout changes long before a full crawl would.
Log every request with its URL, status code and duration. When something goes wrong at two in the morning, those logs are the difference between a quick fix and a long investigation.
Putting it into practice
Treat HTTP 429 responses as a signal, not an obstacle. Read the Retry-After header, back off, and resume at a slower pace.
Be careful with relative URLs. Resolve every link against the page it came from, not against the site root, or you will request pages that do not exist.
Key takeaway
Look for the Sitemap line in robots.txt, follow sitemap index files to their children, and use lastmod to skip pages that have not changed.