Cursor vs offset pagination, part 7
This is part 7 of our series, and this time we focus on edge cases readers asked about. Offset pagination asks for 'items 40 to 60'. Cursor pagination asks for 'the next page after this marker'. The…
Showing 37–48 of 200 articles
This is part 7 of our series, and this time we focus on edge cases readers asked about. Offset pagination asks for 'items 40 to 60'. Cursor pagination asks for 'the next page after this marker'. The…
Infinite scroll replaces the familiar Next button with a page that keeps growing as you move down it, which means your script has to decide when there is nothing left to load.
Part 2 looks at what changes when the same technique has to run every day in production. Logging in once and staying logged in is a matter of keeping the cookies the server gives you and sending them…
For part 3, we revisit the basics with a fresh example and a few lessons learned the hard way. Lazy loading defers images and sections until they are about to scroll into view. A script that reads…
For part 4, we revisit the basics with a fresh example and a few lessons learned the hard way. A one-off script grows into a pipeline the moment someone asks for the data again next week. A little…
In part 5 we take the ideas from earlier instalments and apply them to a larger, messier site. Cross-site request forgery tokens are one of the first things that trip up a script trying to log in to…
This is part 5 of our series, and this time we focus on edge cases readers asked about. Many product pages already contain a clean, structured description of themselves in a script tag meant for…
Part 8 looks at what changes when the same technique has to run every day in production. Infinite scroll replaces the familiar Next button with a page that keeps growing as you move down it, which…
For part 4, we revisit the basics with a fresh example and a few lessons learned the hard way. robots.txt is a short text file with a long history. It tells automated clients which parts of a site…
Part 10 looks at what changes when the same technique has to run every day in production. Logging in once and staying logged in is a matter of keeping the cookies the server gives you and sending…
This is part 9 of our series, and this time we focus on edge cases readers asked about. When the data you want is several levels deep and the markup has few useful attributes, XPath's ability to…
In part 9 we take the ideas from earlier instalments and apply them to a larger, messier site. Scrapy makes it very easy to send thousands of requests a minute. Making a spider that a site operator…