Part 7 looks at what changes when the same technique has to run every day in production. When the data you want is several levels deep and the markup has few useful attributes, XPath's ability to navigate up, down and sideways through the tree becomes very handy.
A practical approach
Pagination comes in several shapes: numbered pages, next links, offsets and opaque cursors. Each has trade-offs, and a robust scraper should detect the end of the result set rather than assuming a fixed number of pages.
Selectors are the most fragile part of any scraper. Prefer stable attributes such as data-testid or semantic structure over generated class names, which can change on every deploy.
Step by step
Time zones matter more than they seem. Store timestamps in UTC and convert only when displaying them.
Write small, focused tests for your parsers using saved HTML fixtures. They run in milliseconds and catch layout changes long before a full crawl would.
Key takeaway
Anchor on a stable nearby element, then use relative paths and axes such as following-sibling and ancestor rather than absolute paths from the root.