XPath for nested structures, part 2
In part 2 we take the ideas from earlier instalments and apply them to a larger, messier site. When the data you want is several levels deep and the markup has few useful attributes, XPath's ability…
Showing 1–12 of 59 articles
In part 2 we take the ideas from earlier instalments and apply them to a larger, messier site. When the data you want is several levels deep and the markup has few useful attributes, XPath's ability…
robots.txt is a short text file with a long history. It tells automated clients which parts of a site the operator would prefer they left alone, and how quickly they should crawl the rest.
Part 2 looks at what changes when the same technique has to run every day in production. Cross-site request forgery tokens are one of the first things that trip up a script trying to log in to a…
For part 3, we revisit the basics with a fresh example and a few lessons learned the hard way. Scrapy makes it very easy to send thousands of requests a minute. Making a spider that a site operator…
Part 8 looks at what changes when the same technique has to run every day in production. A login form is a small piece of UI with a surprising number of paths through it: success, wrong password,…
This is part 7 of our series, and this time we focus on edge cases readers asked about. Offset pagination asks for 'items 40 to 60'. Cursor pagination asks for 'the next page after this marker'. The…
Part 2 looks at what changes when the same technique has to run every day in production. Logging in once and staying logged in is a matter of keeping the cookies the server gives you and sending them…
In part 5 we take the ideas from earlier instalments and apply them to a larger, messier site. Cross-site request forgery tokens are one of the first things that trip up a script trying to log in to…
This is part 5 of our series, and this time we focus on edge cases readers asked about. Many product pages already contain a clean, structured description of themselves in a script tag meant for…
In part 9 we take the ideas from earlier instalments and apply them to a larger, messier site. Scrapy makes it very easy to send thousands of requests a minute. Making a spider that a site operator…
Part 8 looks at what changes when the same technique has to run every day in production. Running a browser without a window is faster and works on servers, but some problems are much easier to…
This is part 7 of our series, and this time we focus on edge cases readers asked about. A well-behaved API tells you how many requests you have left before it starts saying no. Reading those headers…