{"data":{"id":137,"slug":"reading-sitemaps-to-discover-urls-part-7-137","title":"Reading sitemaps to discover URLs, part 7","excerpt":"This is part 7 of our series, and this time we focus on edge cases readers asked about. Before you crawl a site link by link, check whether it already publishes a list of every page it wants found.…","seriesName":"Reading sitemaps to discover URLs","seriesPart":7,"sections":[{"heading":"Introduction","paragraphs":["This is part 7 of our series, and this time we focus on edge cases readers asked about. Before you crawl a site link by link, check whether it already publishes a list of every page it wants found. That list is usually called sitemap.xml."]},{"heading":"What to watch for","paragraphs":["Write small, focused tests for your parsers using saved HTML fixtures. They run in milliseconds and catch layout changes long before a full crawl would.","Log every request with its URL, status code and duration. When something goes wrong at two in the morning, those logs are the difference between a quick fix and a long investigation."]},{"heading":"Putting it into practice","paragraphs":["Treat HTTP 429 responses as a signal, not an obstacle. Read the Retry-After header, back off, and resume at a slower pace.","Be careful with relative URLs. Resolve every link against the page it came from, not against the site root, or you will request pages that do not exist."]},{"heading":"Key takeaway","paragraphs":["Look for the Sitemap line in robots.txt, follow sitemap index files to their children, and use lastmod to skip pages that have not changed."]}],"authorId":6,"authorName":"Sofia Fischer","authorSlug":"sofia-fischer","category":"Scraping Basics","tags":["selenium","playwright","python","ethics"],"publishedAt":"2023-11-26T00:00:00.000Z","wordCount":176,"readingTimeMinutes":2}}