{"data":{"id":17,"slug":"reading-sitemaps-to-discover-urls-17","title":"Reading sitemaps to discover URLs","excerpt":"Before you crawl a site link by link, check whether it already publishes a list of every page it wants found. That list is usually called sitemap.xml.","seriesName":"Reading sitemaps to discover URLs","seriesPart":1,"sections":[{"heading":"Introduction","paragraphs":["Before you crawl a site link by link, check whether it already publishes a list of every page it wants found. That list is usually called sitemap.xml."]},{"heading":"What to watch for","paragraphs":["Always identify your client honestly with a descriptive User-Agent, and keep your request rate modest. A crawler that behaves like a considerate visitor is far less likely to cause problems for the site operator.","Store raw responses alongside parsed output while you are developing. When a parser breaks, you can replay the saved HTML instead of hitting the site again.","Selectors are the most fragile part of any scraper. Prefer stable attributes such as data-testid or semantic structure over generated class names, which can change on every deploy."]},{"heading":"Common pitfalls","paragraphs":["Read the site's terms of service before you start. Many sites welcome automated access within limits and say so explicitly; others ask you to use an API instead.","Keep configuration such as base URLs, delays and selectors out of your code. Changing a delay should not require a redeploy."]},{"heading":"Key takeaway","paragraphs":["Look for the Sitemap line in robots.txt, follow sitemap index files to their children, and use lastmod to skip pages that have not changed."]}],"authorId":8,"authorName":"Hannah Novak","authorSlug":"hannah-novak","category":"Scraping Basics","tags":["selenium","json"],"publishedAt":"2024-07-30T00:00:00.000Z","wordCount":188,"readingTimeMinutes":2}}