Handling infinite scroll in Playwright, part 9
For part 9, we revisit the basics with a fresh example and a few lessons learned the hard way. Infinite scroll replaces the familiar Next button with a page that keeps growing as you move down it,…
Showing 181–192 of 200 articles
For part 9, we revisit the basics with a fresh example and a few lessons learned the hard way. Infinite scroll replaces the familiar Next button with a page that keeps growing as you move down it,…
Part 8 looks at what changes when the same technique has to run every day in production. Most flaky Selenium tests are timing problems in disguise. The element exists, just not yet, and a fixed sleep…
For part 5, we revisit the basics with a fresh example and a few lessons learned the hard way. Most flaky Selenium tests are timing problems in disguise. The element exists, just not yet, and a fixed…
In part 3 we take the ideas from earlier instalments and apply them to a larger, messier site. Infinite scroll replaces the familiar Next button with a page that keeps growing as you move down it,…
This is part 7 of our series, and this time we focus on edge cases readers asked about. Tables look like the easiest thing on a page to scrape, until you meet merged cells, header rows that repeat…
In part 7 we take the ideas from earlier instalments and apply them to a larger, messier site. Running a browser without a window is faster and works on servers, but some problems are much easier to…
Part 2 looks at what changes when the same technique has to run every day in production. Scrapy makes it very easy to send thousands of requests a minute. Making a spider that a site operator would…
Part 5 looks at what changes when the same technique has to run every day in production. CSV is the lingua franca of spreadsheets and the most common way scraped data leaves a project. It is also…
Part 4 looks at what changes when the same technique has to run every day in production. A login form is a small piece of UI with a surprising number of paths through it: success, wrong password,…
In part 4 we take the ideas from earlier instalments and apply them to a larger, messier site. Offset pagination asks for 'items 40 to 60'. Cursor pagination asks for 'the next page after this…
Networks fail, servers restart and rate limits kick in. A scraper that gives up at the first error is fragile; one that retries instantly and forever is a nuisance.
Part 3 looks at what changes when the same technique has to run every day in production. Many product pages already contain a clean, structured description of themselves in a script tag meant for…