Part 4 looks at what changes when the same technique has to run every day in production. A selector that worked yesterday and fails today is usually a selector that depended on something nobody promised to keep the same, such as a generated class name or the position of a div.

Going further

Dynamic pages render content after the initial HTML arrives. A browser automation tool can wait for a specific element to appear, which is far more reliable than sleeping for a fixed amount of time.

Store raw responses alongside parsed output while you are developing. When a parser breaks, you can replay the saved HTML instead of hitting the site again.

Putting it into practice

Selectors are the most fragile part of any scraper. Prefer stable attributes such as data-testid or semantic structure over generated class names, which can change on every deploy.

Pagination comes in several shapes: numbered pages, next links, offsets and opaque cursors. Each has trade-offs, and a robust scraper should detect the end of the result set rather than assuming a fixed number of pages.

Key takeaway

Prefer data-testid attributes, ARIA roles and visible labels. Avoid long chains of nth-child and anything that looks like a hash.