Structuring a data extraction pipeline, part 5
This is part 5 of our series, and this time we focus on edge cases readers asked about. A one-off script grows into a pipeline the moment someone asks for the data again next week. A little structure…
Showing 25–36 of 56 articles
This is part 5 of our series, and this time we focus on edge cases readers asked about. A one-off script grows into a pipeline the moment someone asks for the data again next week. A little structure…
For part 5, we revisit the basics with a fresh example and a few lessons learned the hard way. A selector that worked yesterday and fails today is usually a selector that depended on something nobody…
This is part 5 of our series, and this time we focus on edge cases readers asked about. When the data you want is several levels deep and the markup has few useful attributes, XPath's ability to…
For part 3, we revisit the basics with a fresh example and a few lessons learned the hard way. If a site offers an API for the data you need, scraping its HTML is usually the harder, more fragile and…
In part 4 we take the ideas from earlier instalments and apply them to a larger, messier site. A well-behaved API tells you how many requests you have left before it starts saying no. Reading those…
For part 8, we revisit the basics with a fresh example and a few lessons learned the hard way. A one-off script grows into a pipeline the moment someone asks for the data again next week. A little…
In part 2 we take the ideas from earlier instalments and apply them to a larger, messier site. Networks fail, servers restart and rate limits kick in. A scraper that gives up at the first error is…
In part 3 we take the ideas from earlier instalments and apply them to a larger, messier site. A login form is a small piece of UI with a surprising number of paths through it: success, wrong…
Part 6 looks at what changes when the same technique has to run every day in production. Logging in once and staying logged in is a matter of keeping the cookies the server gives you and sending them…
This is part 4 of our series, and this time we focus on edge cases readers asked about. Logging in once and staying logged in is a matter of keeping the cookies the server gives you and sending them…
In part 9 we take the ideas from earlier instalments and apply them to a larger, messier site. Lazy loading defers images and sections until they are about to scroll into view. A script that reads…
This is part 3 of our series, and this time we focus on edge cases readers asked about. Tables look like the easiest thing on a page to scrape, until you meet merged cells, header rows that repeat…