Part 5 looks at what changes when the same technique has to run every day in production. Offset pagination asks for 'items 40 to 60'. Cursor pagination asks for 'the next page after this marker'. The difference matters as soon as the underlying data changes while you are reading it.

Going further

Pagination comes in several shapes: numbered pages, next links, offsets and opaque cursors. Each has trade-offs, and a robust scraper should detect the end of the result set rather than assuming a fixed number of pages.

Validate the data you extract. Prices without currencies, dates without time zones and empty fields are common sources of silent errors downstream.

Character encodings still cause trouble. Decode responses using the charset the server declares, and normalise text before comparing or storing it.

Common pitfalls

Write small, focused tests for your parsers using saved HTML fixtures. They run in milliseconds and catch layout changes long before a full crawl would.

Selectors are the most fragile part of any scraper. Prefer stable attributes such as data-testid or semantic structure over generated class names, which can change on every deploy.

Treat HTTP 429 responses as a signal, not an obstacle. Read the Retry-After header, back off, and resume at a slower pace.

Key takeaway

Use cursors when the API offers them, treat them as opaque strings, and stop when the API returns no next cursor rather than when a page is short.