When to use an API instead of scraping, part 7
For part 7, we revisit the basics with a fresh example and a few lessons learned the hard way. If a site offers an API for the data you need, scraping its HTML is usually the harder, more fragile and…
Showing 49–60 of 64 articles
For part 7, we revisit the basics with a fresh example and a few lessons learned the hard way. If a site offers an API for the data you need, scraping its HTML is usually the harder, more fragile and…
For part 6, we revisit the basics with a fresh example and a few lessons learned the hard way. Before you crawl a site link by link, check whether it already publishes a list of every page it wants…
For part 4, we revisit the basics with a fresh example and a few lessons learned the hard way. Networks fail, servers restart and rate limits kick in. A scraper that gives up at the first error is…
This is part 4 of our series, and this time we focus on edge cases readers asked about. If a site offers an API for the data you need, scraping its HTML is usually the harder, more fragile and less…
For part 9, we revisit the basics with a fresh example and a few lessons learned the hard way. A selector that worked yesterday and fails today is usually a selector that depended on something nobody…
Part 10 looks at what changes when the same technique has to run every day in production. Scrapy makes it very easy to send thousands of requests a minute. Making a spider that a site operator would…
Part 3 looks at what changes when the same technique has to run every day in production. A one-off script grows into a pipeline the moment someone asks for the data again next week. A little…
In part 4 we take the ideas from earlier instalments and apply them to a larger, messier site. Tables look like the easiest thing on a page to scrape, until you meet merged cells, header rows that…
This is part 3 of our series, and this time we focus on edge cases readers asked about. A well-behaved API tells you how many requests you have left before it starts saying no. Reading those headers…
For part 7, we revisit the basics with a fresh example and a few lessons learned the hard way. Lazy loading defers images and sections until they are about to scroll into view. A script that reads…
Part 8 looks at what changes when the same technique has to run every day in production. Most flaky Selenium tests are timing problems in disguise. The element exists, just not yet, and a fixed sleep…
In part 7 we take the ideas from earlier instalments and apply them to a larger, messier site. Running a browser without a window is faster and works on servers, but some problems are much easier to…