When to use an API instead of scraping, part 4
This is part 4 of our series, and this time we focus on edge cases readers asked about. If a site offers an API for the data you need, scraping its HTML is usually the harder, more fragile and less…
Showing 49–60 of 67 articles
This is part 4 of our series, and this time we focus on edge cases readers asked about. If a site offers an API for the data you need, scraping its HTML is usually the harder, more fragile and less…
This is part 10 of our series, and this time we focus on edge cases readers asked about. Running a browser without a window is faster and works on servers, but some problems are much easier to…
This is part 4 of our series, and this time we focus on edge cases readers asked about. Scrapy makes it very easy to send thousands of requests a minute. Making a spider that a site operator would be…
For part 9, we revisit the basics with a fresh example and a few lessons learned the hard way. A selector that worked yesterday and fails today is usually a selector that depended on something nobody…
In part 8 we take the ideas from earlier instalments and apply them to a larger, messier site. CSV is the lingua franca of spreadsheets and the most common way scraped data leaves a project. It is…
In part 5 we take the ideas from earlier instalments and apply them to a larger, messier site. Scrapy makes it very easy to send thousands of requests a minute. Making a spider that a site operator…
Part 7 looks at what changes when the same technique has to run every day in production. When the data you want is several levels deep and the markup has few useful attributes, XPath's ability to…
In part 5 we take the ideas from earlier instalments and apply them to a larger, messier site. If a site offers an API for the data you need, scraping its HTML is usually the harder, more fragile and…
For part 8, we revisit the basics with a fresh example and a few lessons learned the hard way. robots.txt is a short text file with a long history. It tells automated clients which parts of a site…
For part 5, we revisit the basics with a fresh example and a few lessons learned the hard way. Most flaky Selenium tests are timing problems in disguise. The element exists, just not yet, and a fixed…
In part 3 we take the ideas from earlier instalments and apply them to a larger, messier site. Infinite scroll replaces the familiar Next button with a page that keeps growing as you move down it,…
Part 2 looks at what changes when the same technique has to run every day in production. Scrapy makes it very easy to send thousands of requests a minute. Making a spider that a site operator would…