This is part 3 of our series, and this time we focus on edge cases readers asked about. Offset pagination asks for 'items 40 to 60'. Cursor pagination asks for 'the next page after this marker'. The difference matters as soon as the underlying data changes while you are reading it.

A practical approach

Keep configuration such as base URLs, delays and selectors out of your code. Changing a delay should not require a redeploy.

Read the site's terms of service before you start. Many sites welcome automated access within limits and say so explicitly; others ask you to use an API instead.

Validate the data you extract. Prices without currencies, dates without time zones and empty fields are common sources of silent errors downstream.

Putting it into practice

Character encodings still cause trouble. Decode responses using the charset the server declares, and normalise text before comparing or storing it.

Treat HTTP 429 responses as a signal, not an obstacle. Read the Retry-After header, back off, and resume at a slower pace.

Key takeaway

Use cursors when the API offers them, treat them as opaque strings, and stop when the API returns no next cursor rather than when a page is short.