Building a polite Scrapy spider
Scrapy makes it very easy to send thousands of requests a minute. Making a spider that a site operator would be happy to see in their logs takes a few deliberate settings.
Showing 1–12 of 68 articles
Scrapy makes it very easy to send thousands of requests a minute. Making a spider that a site operator would be happy to see in their logs takes a few deliberate settings.
Part 3 looks at what changes when the same technique has to run every day in production. When the data you want is several levels deep and the markup has few useful attributes, XPath's ability to…
In part 5 we take the ideas from earlier instalments and apply them to a larger, messier site. Lazy loading defers images and sections until they are about to scroll into view. A script that reads…
Part 9 looks at what changes when the same technique has to run every day in production. Tables look like the easiest thing on a page to scrape, until you meet merged cells, header rows that repeat…
In part 2 we take the ideas from earlier instalments and apply them to a larger, messier site. robots.txt is a short text file with a long history. It tells automated clients which parts of a site…
Part 6 looks at what changes when the same technique has to run every day in production. Scrapy makes it very easy to send thousands of requests a minute. Making a spider that a site operator would…
In part 9 we take the ideas from earlier instalments and apply them to a larger, messier site. Cross-site request forgery tokens are one of the first things that trip up a script trying to log in to…
In part 6 we take the ideas from earlier instalments and apply them to a larger, messier site. Networks fail, servers restart and rate limits kick in. A scraper that gives up at the first error is…
This is part 2 of our series, and this time we focus on edge cases readers asked about. Running a browser without a window is faster and works on servers, but some problems are much easier to…
Part 6 looks at what changes when the same technique has to run every day in production. Lazy loading defers images and sections until they are about to scroll into view. A script that reads the page…
Part 4 looks at what changes when the same technique has to run every day in production. A selector that worked yesterday and fails today is usually a selector that depended on something nobody…
A one-off script grows into a pipeline the moment someone asks for the data again next week. A little structure early on saves a rewrite later.