XPath for nested structures, part 2
In part 2 we take the ideas from earlier instalments and apply them to a larger, messier site. When the data you want is several levels deep and the markup has few useful attributes, XPath's ability…
Showing 1–12 of 67 articles
In part 2 we take the ideas from earlier instalments and apply them to a larger, messier site. When the data you want is several levels deep and the markup has few useful attributes, XPath's ability…
Part 3 looks at what changes when the same technique has to run every day in production. When the data you want is several levels deep and the markup has few useful attributes, XPath's ability to…
Part 9 looks at what changes when the same technique has to run every day in production. Tables look like the easiest thing on a page to scrape, until you meet merged cells, header rows that repeat…
In part 7 we take the ideas from earlier instalments and apply them to a larger, messier site. A login form is a small piece of UI with a surprising number of paths through it: success, wrong…
In part 6 we take the ideas from earlier instalments and apply them to a larger, messier site. Many product pages already contain a clean, structured description of themselves in a script tag meant…
In part 9 we take the ideas from earlier instalments and apply them to a larger, messier site. Cross-site request forgery tokens are one of the first things that trip up a script trying to log in to…
For part 7, we revisit the basics with a fresh example and a few lessons learned the hard way. Cross-site request forgery tokens are one of the first things that trip up a script trying to log in to…
A one-off script grows into a pipeline the moment someone asks for the data again next week. A little structure early on saves a rewrite later.
This is part 2 of our series, and this time we focus on edge cases readers asked about. Infinite scroll replaces the familiar Next button with a page that keeps growing as you move down it, which…
In part 8 we take the ideas from earlier instalments and apply them to a larger, messier site. Before you crawl a site link by link, check whether it already publishes a list of every page it wants…
This is part 7 of our series, and this time we focus on edge cases readers asked about. Offset pagination asks for 'items 40 to 60'. Cursor pagination asks for 'the next page after this marker'. The…
Infinite scroll replaces the familiar Next button with a page that keeps growing as you move down it, which means your script has to decide when there is nothing left to load.