Handling infinite scroll in Playwright, part 5
For part 5, we revisit the basics with a fresh example and a few lessons learned the hard way. Infinite scroll replaces the familiar Next button with a page that keeps growing as you move down it,…
Showing 25–36 of 64 articles
For part 5, we revisit the basics with a fresh example and a few lessons learned the hard way. Infinite scroll replaces the familiar Next button with a page that keeps growing as you move down it,…
Tables look like the easiest thing on a page to scrape, until you meet merged cells, header rows that repeat every twenty lines and numbers formatted for humans rather than machines.
In part 10 we take the ideas from earlier instalments and apply them to a larger, messier site. robots.txt is a short text file with a long history. It tells automated clients which parts of a site…
Part 2 looks at what changes when the same technique has to run every day in production. If a site offers an API for the data you need, scraping its HTML is usually the harder, more fragile and less…
This is part 10 of our series, and this time we focus on edge cases readers asked about. Most flaky Selenium tests are timing problems in disguise. The element exists, just not yet, and a fixed sleep…
For part 7, we revisit the basics with a fresh example and a few lessons learned the hard way. Scrapy makes it very easy to send thousands of requests a minute. Making a spider that a site operator…
For part 4, we revisit the basics with a fresh example and a few lessons learned the hard way. When the data you want is several levels deep and the markup has few useful attributes, XPath's ability…
For part 3, we revisit the basics with a fresh example and a few lessons learned the hard way. If a site offers an API for the data you need, scraping its HTML is usually the harder, more fragile and…
This is part 6 of our series, and this time we focus on edge cases readers asked about. Most flaky Selenium tests are timing problems in disguise. The element exists, just not yet, and a fixed sleep…
Part 6 looks at what changes when the same technique has to run every day in production. Cross-site request forgery tokens are one of the first things that trip up a script trying to log in to a…
Part 4 looks at what changes when the same technique has to run every day in production. Most flaky Selenium tests are timing problems in disguise. The element exists, just not yet, and a fixed sleep…
Part 9 looks at what changes when the same technique has to run every day in production. Before you crawl a site link by link, check whether it already publishes a list of every page it wants found.…