<?xml version="1.0" encoding="UTF-8"?>
<response><data><post><id>65</id><slug>cursor-vs-offset-pagination-part-4-65</slug><title>Cursor vs offset pagination, part 4</title><excerpt>In part 4 we take the ideas from earlier instalments and apply them to a larger, messier site. Offset pagination asks for 'items 40 to 60'. Cursor pagination asks for 'the next page after this…</excerpt><seriesName>Cursor vs offset pagination</seriesName><seriesPart>4</seriesPart><sections><item><heading>Introduction</heading><paragraphs><item>In part 4 we take the ideas from earlier instalments and apply them to a larger, messier site. Offset pagination asks for 'items 40 to 60'. Cursor pagination asks for 'the next page after this marker'. The difference matters as soon as the underlying data changes while you are reading it.</item></paragraphs></item><item><heading>Putting it into practice</heading><paragraphs><item>Time zones matter more than they seem. Store timestamps in UTC and convert only when displaying them.</item><item>Store raw responses alongside parsed output while you are developing. When a parser breaks, you can replay the saved HTML instead of hitting the site again.</item><item>Deduplicate as you go. The same item often appears on several listing pages, and a stable identifier such as a SKU or ID makes duplicates easy to spot.</item></paragraphs></item><item><heading>Common pitfalls</heading><paragraphs><item>Dynamic pages render content after the initial HTML arrives. A browser automation tool can wait for a specific element to appear, which is far more reliable than sleeping for a fixed amount of time.</item><item>Be careful with relative URLs. Resolve every link against the page it came from, not against the site root, or you will request pages that do not exist.</item></paragraphs></item><item><heading>Key takeaway</heading><paragraphs><item>Use cursors when the API offers them, treat them as opaque strings, and stop when the API returns no next cursor rather than when a page is short.</item></paragraphs></item></sections><authorId>9</authorId><authorName>Mateo Hughes</authorName><authorSlug>mateo-hughes</authorSlug><category>APIs</category><tags><item>python</item><item>testing</item><item>javascript</item><item>http</item></tags><publishedAt>2023-12-29T00:00:00.000Z</publishedAt><wordCount>212</wordCount><readingTimeMinutes>2</readingTimeMinutes></post></data></response>
