XPath for nested structures, part 4
For part 4, we revisit the basics with a fresh example and a few lessons learned the hard way. When the data you want is several levels deep and the markup has few useful attributes, XPath's ability…
Showing 25–36 of 67 articles
For part 4, we revisit the basics with a fresh example and a few lessons learned the hard way. When the data you want is several levels deep and the markup has few useful attributes, XPath's ability…
Many product pages already contain a clean, structured description of themselves in a script tag meant for search engines. Reading it is often more reliable than parsing the visible HTML.
In part 2 we take the ideas from earlier instalments and apply them to a larger, messier site. Networks fail, servers restart and rate limits kick in. A scraper that gives up at the first error is…
Part 4 looks at what changes when the same technique has to run every day in production. Most flaky Selenium tests are timing problems in disguise. The element exists, just not yet, and a fixed sleep…
Part 6 looks at what changes when the same technique has to run every day in production. Logging in once and staying logged in is a matter of keeping the cookies the server gives you and sending them…
This is part 4 of our series, and this time we focus on edge cases readers asked about. Logging in once and staying logged in is a matter of keeping the cookies the server gives you and sending them…
This is part 9 of our series, and this time we focus on edge cases readers asked about. Many product pages already contain a clean, structured description of themselves in a script tag meant for…
In part 6 we take the ideas from earlier instalments and apply them to a larger, messier site. robots.txt is a short text file with a long history. It tells automated clients which parts of a site…
Cross-site request forgery tokens are one of the first things that trip up a script trying to log in to a site: the form posts fine in a browser but fails with a 403 from code.
In part 7 we take the ideas from earlier instalments and apply them to a larger, messier site. Most flaky Selenium tests are timing problems in disguise. The element exists, just not yet, and a fixed…
In part 10 we take the ideas from earlier instalments and apply them to a larger, messier site. When the data you want is several levels deep and the markup has few useful attributes, XPath's ability…
This is part 4 of our series, and this time we focus on edge cases readers asked about. Cross-site request forgery tokens are one of the first things that trip up a script trying to log in to a site:…