Respecting robots.txt in practice
robots.txt is a short text file with a long history. It tells automated clients which parts of a site the operator would prefer they left alone, and how quickly they should crawl the rest.
QA Engineer
Noah writes about practical data extraction and test automation.
15 articles
robots.txt is a short text file with a long history. It tells automated clients which parts of a site the operator would prefer they left alone, and how quickly they should crawl the rest.
Part 6 looks at what changes when the same technique has to run every day in production. Scrapy makes it very easy to send thousands of requests a minute. Making a spider that a site operator would…
For part 7, we revisit the basics with a fresh example and a few lessons learned the hard way. Cross-site request forgery tokens are one of the first things that trip up a script trying to log in to…
A one-off script grows into a pipeline the moment someone asks for the data again next week. A little structure early on saves a rewrite later.
For part 9, we revisit the basics with a fresh example and a few lessons learned the hard way. Running a browser without a window is faster and works on servers, but some problems are much easier to…
Tables look like the easiest thing on a page to scrape, until you meet merged cells, header rows that repeat every twenty lines and numbers formatted for humans rather than machines.
This is part 5 of our series, and this time we focus on edge cases readers asked about. A one-off script grows into a pipeline the moment someone asks for the data again next week. A little structure…
In part 9 we take the ideas from earlier instalments and apply them to a larger, messier site. Logging in once and staying logged in is a matter of keeping the cookies the server gives you and…
This is part 2 of our series, and this time we focus on edge cases readers asked about. Most flaky Selenium tests are timing problems in disguise. The element exists, just not yet, and a fixed sleep…
Part 2 looks at what changes when the same technique has to run every day in production. Lazy loading defers images and sections until they are about to scroll into view. A script that reads the page…
In part 8 we take the ideas from earlier instalments and apply them to a larger, messier site. CSV is the lingua franca of spreadsheets and the most common way scraped data leaves a project. It is…
For part 2, we revisit the basics with a fresh example and a few lessons learned the hard way. Before you crawl a site link by link, check whether it already publishes a list of every page it wants…
This is part 3 of our series, and this time we focus on edge cases readers asked about. Before you crawl a site link by link, check whether it already publishes a list of every page it wants found.…
For part 7, we revisit the basics with a fresh example and a few lessons learned the hard way. Lazy loading defers images and sections until they are about to scroll into view. A script that reads…
In part 7 we take the ideas from earlier instalments and apply them to a larger, messier site. Running a browser without a window is faster and works on servers, but some problems are much easier to…