Building a polite Scrapy spider
Scrapy makes it very easy to send thousands of requests a minute. Making a spider that a site operator would be happy to see in their logs takes a few deliberate settings.
Developer Advocate
Hannah writes about practical data extraction and test automation.
19 articles
Scrapy makes it very easy to send thousands of requests a minute. Making a spider that a site operator would be happy to see in their logs takes a few deliberate settings.
In part 7 we take the ideas from earlier instalments and apply them to a larger, messier site. A login form is a small piece of UI with a surprising number of paths through it: success, wrong…
Part 8 looks at what changes when the same technique has to run every day in production. A login form is a small piece of UI with a surprising number of paths through it: success, wrong password,…
In part 9 we take the ideas from earlier instalments and apply them to a larger, messier site. Scrapy makes it very easy to send thousands of requests a minute. Making a spider that a site operator…
Part 8 looks at what changes when the same technique has to run every day in production. Running a browser without a window is faster and works on servers, but some problems are much easier to…
For part 6, we revisit the basics with a fresh example and a few lessons learned the hard way. A well-behaved API tells you how many requests you have left before it starts saying no. Reading those…
In part 10 we take the ideas from earlier instalments and apply them to a larger, messier site. A one-off script grows into a pipeline the moment someone asks for the data again next week. A little…
For part 2, we revisit the basics with a fresh example and a few lessons learned the hard way. Offset pagination asks for 'items 40 to 60'. Cursor pagination asks for 'the next page after this…
Part 8 looks at what changes when the same technique has to run every day in production. A selector that worked yesterday and fails today is usually a selector that depended on something nobody…
Part 5 looks at what changes when the same technique has to run every day in production. A well-behaved API tells you how many requests you have left before it starts saying no. Reading those headers…
This is part 8 of our series, and this time we focus on edge cases readers asked about. Scrapy makes it very easy to send thousands of requests a minute. Making a spider that a site operator would be…
Before you crawl a site link by link, check whether it already publishes a list of every page it wants found. That list is usually called sitemap.xml.
For part 7, we revisit the basics with a fresh example and a few lessons learned the hard way. If a site offers an API for the data you need, scraping its HTML is usually the harder, more fragile and…
For part 6, we revisit the basics with a fresh example and a few lessons learned the hard way. Before you crawl a site link by link, check whether it already publishes a list of every page it wants…
In part 4 we take the ideas from earlier instalments and apply them to a larger, messier site. Tables look like the easiest thing on a page to scrape, until you meet merged cells, header rows that…
Part 8 looks at what changes when the same technique has to run every day in production. Most flaky Selenium tests are timing problems in disguise. The element exists, just not yet, and a fixed sleep…
For part 5, we revisit the basics with a fresh example and a few lessons learned the hard way. Most flaky Selenium tests are timing problems in disguise. The element exists, just not yet, and a fixed…
Part 4 looks at what changes when the same technique has to run every day in production. A login form is a small piece of UI with a surprising number of paths through it: success, wrong password,…
Part 3 looks at what changes when the same technique has to run every day in production. Many product pages already contain a clean, structured description of themselves in a script tag meant for…