Headless vs headed browsers, part 4
Part 4 looks at what changes when the same technique has to run every day in production. Running a browser without a window is faster and works on servers, but some problems are much easier to…
Showing 25–36 of 68 articles
Part 4 looks at what changes when the same technique has to run every day in production. Running a browser without a window is faster and works on servers, but some problems are much easier to…
For part 6, we revisit the basics with a fresh example and a few lessons learned the hard way. Offset pagination asks for 'items 40 to 60'. Cursor pagination asks for 'the next page after this…
In part 10 we take the ideas from earlier instalments and apply them to a larger, messier site. Many product pages already contain a clean, structured description of themselves in a script tag meant…
This is part 4 of our series, and this time we focus on edge cases readers asked about. Lazy loading defers images and sections until they are about to scroll into view. A script that reads the page…
For part 9, we revisit the basics with a fresh example and a few lessons learned the hard way. Running a browser without a window is faster and works on servers, but some problems are much easier to…
In part 8 we take the ideas from earlier instalments and apply them to a larger, messier site. Offset pagination asks for 'items 40 to 60'. Cursor pagination asks for 'the next page after this…
For part 3, we revisit the basics with a fresh example and a few lessons learned the hard way. Cross-site request forgery tokens are one of the first things that trip up a script trying to log in to…
This is part 10 of our series, and this time we focus on edge cases readers asked about. Most flaky Selenium tests are timing problems in disguise. The element exists, just not yet, and a fixed sleep…
For part 3, we revisit the basics with a fresh example and a few lessons learned the hard way. If a site offers an API for the data you need, scraping its HTML is usually the harder, more fragile and…
This is part 6 of our series, and this time we focus on edge cases readers asked about. Most flaky Selenium tests are timing problems in disguise. The element exists, just not yet, and a fixed sleep…
For part 8, we revisit the basics with a fresh example and a few lessons learned the hard way. Networks fail, servers restart and rate limits kick in. A scraper that gives up at the first error is…
For part 8, we revisit the basics with a fresh example and a few lessons learned the hard way. A one-off script grows into a pipeline the moment someone asks for the data again next week. A little…