Respecting robots.txt in practice, part 2
In part 2 we take the ideas from earlier instalments and apply them to a larger, messier site. robots.txt is a short text file with a long history. It tells automated clients which parts of a site…
Developer Advocate
Kwame writes about practical data extraction and test automation.
13 articles
In part 2 we take the ideas from earlier instalments and apply them to a larger, messier site. robots.txt is a short text file with a long history. It tells automated clients which parts of a site…
In part 6 we take the ideas from earlier instalments and apply them to a larger, messier site. Networks fail, servers restart and rate limits kick in. A scraper that gives up at the first error is…
This is part 7 of our series, and this time we focus on edge cases readers asked about. A well-behaved API tells you how many requests you have left before it starts saying no. Reading those headers…
In part 4 we take the ideas from earlier instalments and apply them to a larger, messier site. CSV is the lingua franca of spreadsheets and the most common way scraped data leaves a project. It is…
This is part 4 of our series, and this time we focus on edge cases readers asked about. Lazy loading defers images and sections until they are about to scroll into view. A script that reads the page…
For part 6, we revisit the basics with a fresh example and a few lessons learned the hard way. Tables look like the easiest thing on a page to scrape, until you meet merged cells, header rows that…
For part 3, we revisit the basics with a fresh example and a few lessons learned the hard way. Cross-site request forgery tokens are one of the first things that trip up a script trying to log in to…
This is part 10 of our series, and this time we focus on edge cases readers asked about. Most flaky Selenium tests are timing problems in disguise. The element exists, just not yet, and a fixed sleep…
In part 3 we take the ideas from earlier instalments and apply them to a larger, messier site. A login form is a small piece of UI with a surprising number of paths through it: success, wrong…
Part 10 looks at what changes when the same technique has to run every day in production. Cross-site request forgery tokens are one of the first things that trip up a script trying to log in to a…
Lazy loading defers images and sections until they are about to scroll into view. A script that reads the page immediately sees placeholders instead of content.
Part 7 looks at what changes when the same technique has to run every day in production. A one-off script grows into a pipeline the moment someone asks for the data again next week. A little…
A selector that worked yesterday and fails today is usually a selector that depended on something nobody promised to keep the same, such as a generated class name or the position of a div.