This is part 9 of our series, and this time we focus on edge cases readers asked about. When the data you want is several levels deep and the markup has few useful attributes, XPath's ability to navigate up, down and sideways through the tree becomes very handy.
What to watch for
Time zones matter more than they seem. Store timestamps in UTC and convert only when displaying them.
Store raw responses alongside parsed output while you are developing. When a parser breaks, you can replay the saved HTML instead of hitting the site again.
Keep configuration such as base URLs, delays and selectors out of your code. Changing a delay should not require a redeploy.
A practical approach
Write small, focused tests for your parsers using saved HTML fixtures. They run in milliseconds and catch layout changes long before a full crawl would.
Pagination comes in several shapes: numbered pages, next links, offsets and opaque cursors. Each has trade-offs, and a robust scraper should detect the end of the result set rather than assuming a fixed number of pages.
Key takeaway
Anchor on a stable nearby element, then use relative paths and axes such as following-sibling and ancestor rather than absolute paths from the root.