<?xml version="1.0" encoding="UTF-8"?>
<response><data><post><id>191</id><slug>xpath-for-nested-structures-part-10-191</slug><title>XPath for nested structures, part 10</title><excerpt>In part 10 we take the ideas from earlier instalments and apply them to a larger, messier site. When the data you want is several levels deep and the markup has few useful attributes, XPath's ability…</excerpt><seriesName>XPath for nested structures</seriesName><seriesPart>10</seriesPart><sections><item><heading>Introduction</heading><paragraphs><item>In part 10 we take the ideas from earlier instalments and apply them to a larger, messier site. When the data you want is several levels deep and the markup has few useful attributes, XPath's ability to navigate up, down and sideways through the tree becomes very handy.</item></paragraphs></item><item><heading>Step by step</heading><paragraphs><item>Log every request with its URL, status code and duration. When something goes wrong at two in the morning, those logs are the difference between a quick fix and a long investigation.</item><item>Keep configuration such as base URLs, delays and selectors out of your code. Changing a delay should not require a redeploy.</item><item>Validate the data you extract. Prices without currencies, dates without time zones and empty fields are common sources of silent errors downstream.</item></paragraphs></item><item><heading>What to watch for</heading><paragraphs><item>Selectors are the most fragile part of any scraper. Prefer stable attributes such as data-testid or semantic structure over generated class names, which can change on every deploy.</item><item>Pagination comes in several shapes: numbered pages, next links, offsets and opaque cursors. Each has trade-offs, and a robust scraper should detect the end of the result set rather than assuming a fixed number of pages.</item><item>Deduplicate as you go. The same item often appears on several listing pages, and a stable identifier such as a SKU or ID makes duplicates easy to spot.</item></paragraphs></item><item><heading>Key takeaway</heading><paragraphs><item>Anchor on a stable nearby element, then use relative paths and axes such as following-sibling and ancestor rather than absolute paths from the root.</item></paragraphs></item></sections><authorId>10</authorId><authorName>Aisha Rossi</authorName><authorSlug>aisha-rossi</authorSlug><category>Scraping Basics</category><tags><item>ethics</item><item>python</item><item>selenium</item></tags><publishedAt>2025-01-30T00:00:00.000Z</publishedAt><wordCount>239</wordCount><readingTimeMinutes>2</readingTimeMinutes></post></data></response>
