Detecting lazy-loaded content, part 5
In part 5 we take the ideas from earlier instalments and apply them to a larger, messier site. Lazy loading defers images and sections until they are about to scroll into view. A script that reads…
Data Engineer
Mateo writes about practical data extraction and test automation.
18 articles
In part 5 we take the ideas from earlier instalments and apply them to a larger, messier site. Lazy loading defers images and sections until they are about to scroll into view. A script that reads…
Part 2 looks at what changes when the same technique has to run every day in production. Cross-site request forgery tokens are one of the first things that trip up a script trying to log in to a…
Part 8 looks at what changes when the same technique has to run every day in production. Infinite scroll replaces the familiar Next button with a page that keeps growing as you move down it, which…
Part 5 looks at what changes when the same technique has to run every day in production. Tables look like the easiest thing on a page to scrape, until you meet merged cells, header rows that repeat…
Part 4 looks at what changes when the same technique has to run every day in production. Most flaky Selenium tests are timing problems in disguise. The element exists, just not yet, and a fixed sleep…
This is part 9 of our series, and this time we focus on edge cases readers asked about. Many product pages already contain a clean, structured description of themselves in a script tag meant for…
In part 9 we take the ideas from earlier instalments and apply them to a larger, messier site. Lazy loading defers images and sections until they are about to scroll into view. A script that reads…
In part 3 we take the ideas from earlier instalments and apply them to a larger, messier site. Most flaky Selenium tests are timing problems in disguise. The element exists, just not yet, and a fixed…
Most flaky Selenium tests are timing problems in disguise. The element exists, just not yet, and a fixed sleep is either too short on a slow day or wastes time on a fast one.
For part 10, we revisit the basics with a fresh example and a few lessons learned the hard way. CSV is the lingua franca of spreadsheets and the most common way scraped data leaves a project. It is…
In part 9 we take the ideas from earlier instalments and apply them to a larger, messier site. If a site offers an API for the data you need, scraping its HTML is usually the harder, more fragile and…
In part 10 we take the ideas from earlier instalments and apply them to a larger, messier site. Networks fail, servers restart and rate limits kick in. A scraper that gives up at the first error is…
This is part 4 of our series, and this time we focus on edge cases readers asked about. If a site offers an API for the data you need, scraping its HTML is usually the harder, more fragile and less…
This is part 4 of our series, and this time we focus on edge cases readers asked about. Scrapy makes it very easy to send thousands of requests a minute. Making a spider that a site operator would be…
In part 5 we take the ideas from earlier instalments and apply them to a larger, messier site. If a site offers an API for the data you need, scraping its HTML is usually the harder, more fragile and…
For part 9, we revisit the basics with a fresh example and a few lessons learned the hard way. Infinite scroll replaces the familiar Next button with a page that keeps growing as you move down it,…
This is part 7 of our series, and this time we focus on edge cases readers asked about. Tables look like the easiest thing on a page to scrape, until you meet merged cells, header rows that repeat…
In part 4 we take the ideas from earlier instalments and apply them to a larger, messier site. Offset pagination asks for 'items 40 to 60'. Cursor pagination asks for 'the next page after this…