In part 8 we take the ideas from earlier instalments and apply them to a larger, messier site. Offset pagination asks for 'items 40 to 60'. Cursor pagination asks for 'the next page after this marker'. The difference matters as soon as the underlying data changes while you are reading it.

Step by step

Write small, focused tests for your parsers using saved HTML fixtures. They run in milliseconds and catch layout changes long before a full crawl would.

Validate the data you extract. Prices without currencies, dates without time zones and empty fields are common sources of silent errors downstream.

Before writing a single line of code, look at how the page actually loads its data. Open the network panel, reload, and note which requests return the content you care about. Very often the data you want arrives as JSON from an endpoint you can call directly.

Common pitfalls

Selectors are the most fragile part of any scraper. Prefer stable attributes such as data-testid or semantic structure over generated class names, which can change on every deploy.

Store raw responses alongside parsed output while you are developing. When a parser breaks, you can replay the saved HTML instead of hitting the site again.

Key takeaway

Use cursors when the API offers them, treat them as opaque strings, and stop when the API returns no next cursor rather than when a page is short.