For part 2, we revisit the basics with a fresh example and a few lessons learned the hard way. Offset pagination asks for 'items 40 to 60'. Cursor pagination asks for 'the next page after this marker'. The difference matters as soon as the underlying data changes while you are reading it.
Going further
Log every request with its URL, status code and duration. When something goes wrong at two in the morning, those logs are the difference between a quick fix and a long investigation.
Always identify your client honestly with a descriptive User-Agent, and keep your request rate modest. A crawler that behaves like a considerate visitor is far less likely to cause problems for the site operator.
Character encodings still cause trouble. Decode responses using the charset the server declares, and normalise text before comparing or storing it.
What to watch for
Treat HTTP 429 responses as a signal, not an obstacle. Read the Retry-After header, back off, and resume at a slower pace.
Before writing a single line of code, look at how the page actually loads its data. Open the network panel, reload, and note which requests return the content you care about. Very often the data you want arrives as JSON from an endpoint you can call directly.
Key takeaway
Use cursors when the API offers them, treat them as opaque strings, and stop when the API returns no next cursor rather than when a page is short.