<?xml version="1.0" encoding="UTF-8"?>
<response><data><post><id>117</id><slug>reading-sitemaps-to-discover-urls-part-6-117</slug><title>Reading sitemaps to discover URLs, part 6</title><excerpt>For part 6, we revisit the basics with a fresh example and a few lessons learned the hard way. Before you crawl a site link by link, check whether it already publishes a list of every page it wants…</excerpt><seriesName>Reading sitemaps to discover URLs</seriesName><seriesPart>6</seriesPart><sections><item><heading>Introduction</heading><paragraphs><item>For part 6, we revisit the basics with a fresh example and a few lessons learned the hard way. Before you crawl a site link by link, check whether it already publishes a list of every page it wants found. That list is usually called sitemap.xml.</item></paragraphs></item><item><heading>Putting it into practice</heading><paragraphs><item>Dynamic pages render content after the initial HTML arrives. A browser automation tool can wait for a specific element to appear, which is far more reliable than sleeping for a fixed amount of time.</item><item>Selectors are the most fragile part of any scraper. Prefer stable attributes such as data-testid or semantic structure over generated class names, which can change on every deploy.</item><item>Log every request with its URL, status code and duration. When something goes wrong at two in the morning, those logs are the difference between a quick fix and a long investigation.</item></paragraphs></item><item><heading>Common pitfalls</heading><paragraphs><item>Before writing a single line of code, look at how the page actually loads its data. Open the network panel, reload, and note which requests return the content you care about. Very often the data you want arrives as JSON from an endpoint you can call directly.</item><item>Keep configuration such as base URLs, delays and selectors out of your code. Changing a delay should not require a redeploy.</item></paragraphs></item><item><heading>Key takeaway</heading><paragraphs><item>Look for the Sitemap line in robots.txt, follow sitemap index files to their children, and use lastmod to skip pages that have not changed.</item></paragraphs></item></sections><authorId>8</authorId><authorName>Hannah Novak</authorName><authorSlug>hannah-novak</authorSlug><category>Scraping Basics</category><tags><item>scrapy</item><item>javascript</item><item>playwright</item></tags><publishedAt>2024-07-10T00:00:00.000Z</publishedAt><wordCount>232</wordCount><readingTimeMinutes>2</readingTimeMinutes></post></data></response>
