Handling infinite scroll in Playwright, part 4
Part 4 looks at what changes when the same technique has to run every day in production. Infinite scroll replaces the familiar Next button with a page that keeps growing as you move down it, which…
Showing 193–200 of 200 articles
Part 4 looks at what changes when the same technique has to run every day in production. Infinite scroll replaces the familiar Next button with a page that keeps growing as you move down it, which…
This is part 3 of our series, and this time we focus on edge cases readers asked about. CSV is the lingua franca of spreadsheets and the most common way scraped data leaves a project. It is also…
In part 5 we take the ideas from earlier instalments and apply them to a larger, messier site. Logging in once and staying logged in is a matter of keeping the cookies the server gives you and…
This is part 7 of our series, and this time we focus on edge cases readers asked about. Before you crawl a site link by link, check whether it already publishes a list of every page it wants found.…
A selector that worked yesterday and fails today is usually a selector that depended on something nobody promised to keep the same, such as a generated class name or the position of a div.
Part 3 looks at what changes when the same technique has to run every day in production. Networks fail, servers restart and rate limits kick in. A scraper that gives up at the first error is fragile;…
In part 4 we take the ideas from earlier instalments and apply them to a larger, messier site. Before you crawl a site link by link, check whether it already publishes a list of every page it wants…
For part 5, we revisit the basics with a fresh example and a few lessons learned the hard way. Running a browser without a window is faster and works on servers, but some problems are much easier to…