In part 7 we take the ideas from earlier instalments and apply them to a larger, messier site. A selector that worked yesterday and fails today is usually a selector that depended on something nobody promised to keep the same, such as a generated class name or the position of a div.
A practical approach
Validate the data you extract. Prices without currencies, dates without time zones and empty fields are common sources of silent errors downstream.
Pagination comes in several shapes: numbered pages, next links, offsets and opaque cursors. Each has trade-offs, and a robust scraper should detect the end of the result set rather than assuming a fixed number of pages.
Common pitfalls
Read the site's terms of service before you start. Many sites welcome automated access within limits and say so explicitly; others ask you to use an API instead.
Selectors are the most fragile part of any scraper. Prefer stable attributes such as data-testid or semantic structure over generated class names, which can change on every deploy.
Key takeaway
Prefer data-testid attributes, ARIA roles and visible labels. Avoid long chains of nth-child and anything that looks like a hash.