{"data":{"id":197,"slug":"reading-sitemaps-to-discover-urls-part-10-197","title":"Reading sitemaps to discover URLs, part 10","excerpt":"For part 10, we revisit the basics with a fresh example and a few lessons learned the hard way. Before you crawl a site link by link, check whether it already publishes a list of every page it wants…","seriesName":"Reading sitemaps to discover URLs","seriesPart":10,"sections":[{"heading":"Introduction","paragraphs":["For part 10, we revisit the basics with a fresh example and a few lessons learned the hard way. Before you crawl a site link by link, check whether it already publishes a list of every page it wants found. That list is usually called sitemap.xml."]},{"heading":"Going further","paragraphs":["Keep configuration such as base URLs, delays and selectors out of your code. Changing a delay should not require a redeploy.","Store raw responses alongside parsed output while you are developing. When a parser breaks, you can replay the saved HTML instead of hitting the site again."]},{"heading":"Step by step","paragraphs":["Write small, focused tests for your parsers using saved HTML fixtures. They run in milliseconds and catch layout changes long before a full crawl would.","Character encodings still cause trouble. Decode responses using the charset the server declares, and normalise text before comparing or storing it."]},{"heading":"Key takeaway","paragraphs":["Look for the Sitemap line in robots.txt, follow sitemap index files to their children, and use lastmod to skip pages that have not changed."]}],"authorId":12,"authorName":"Yuki Andersson","authorSlug":"yuki-andersson","category":"Scraping Basics","tags":["http","json"],"publishedAt":"2026-05-12T00:00:00.000Z","wordCount":163,"readingTimeMinutes":2}}