Headless vs headed browsers, part 2
This is part 2 of our series, and this time we focus on edge cases readers asked about. Running a browser without a window is faster and works on servers, but some problems are much easier to…
Data Engineer
Priya writes about practical data extraction and test automation.
22 articles
This is part 2 of our series, and this time we focus on edge cases readers asked about. Running a browser without a window is faster and works on servers, but some problems are much easier to…
Part 6 looks at what changes when the same technique has to run every day in production. Lazy loading defers images and sections until they are about to scroll into view. A script that reads the page…
This is part 2 of our series, and this time we focus on edge cases readers asked about. Infinite scroll replaces the familiar Next button with a page that keeps growing as you move down it, which…
In part 8 we take the ideas from earlier instalments and apply them to a larger, messier site. Before you crawl a site link by link, check whether it already publishes a list of every page it wants…
This is part 5 of our series, and this time we focus on edge cases readers asked about. Many product pages already contain a clean, structured description of themselves in a script tag meant for…
For part 4, we revisit the basics with a fresh example and a few lessons learned the hard way. robots.txt is a short text file with a long history. It tells automated clients which parts of a site…
For part 9, we revisit the basics with a fresh example and a few lessons learned the hard way. A login form is a small piece of UI with a surprising number of paths through it: success, wrong…
This is part 6 of our series, and this time we focus on edge cases readers asked about. A selector that worked yesterday and fails today is usually a selector that depended on something nobody…
For part 6, we revisit the basics with a fresh example and a few lessons learned the hard way. Offset pagination asks for 'items 40 to 60'. Cursor pagination asks for 'the next page after this…
Part 6 looks at what changes when the same technique has to run every day in production. Logging in once and staying logged in is a matter of keeping the cookies the server gives you and sending them…
This is part 4 of our series, and this time we focus on edge cases readers asked about. Logging in once and staying logged in is a matter of keeping the cookies the server gives you and sending them…
In part 6 we take the ideas from earlier instalments and apply them to a larger, messier site. When the data you want is several levels deep and the markup has few useful attributes, XPath's ability…
In part 6 we take the ideas from earlier instalments and apply them to a larger, messier site. robots.txt is a short text file with a long history. It tells automated clients which parts of a site…
Cross-site request forgery tokens are one of the first things that trip up a script trying to log in to a site: the form posts fine in a browser but fails with a 403 from code.
In part 3 we take the ideas from earlier instalments and apply them to a larger, messier site. A selector that worked yesterday and fails today is usually a selector that depended on something nobody…
This is part 4 of our series, and this time we focus on edge cases readers asked about. Cross-site request forgery tokens are one of the first things that trip up a script trying to log in to a site:…
This is part 5 of our series, and this time we focus on edge cases readers asked about. Networks fail, servers restart and rate limits kick in. A scraper that gives up at the first error is fragile;…
This is part 8 of our series, and this time we focus on edge cases readers asked about. Cross-site request forgery tokens are one of the first things that trip up a script trying to log in to a site:…
For part 2, we revisit the basics with a fresh example and a few lessons learned the hard way. Tables look like the easiest thing on a page to scrape, until you meet merged cells, header rows that…
Part 7 looks at what changes when the same technique has to run every day in production. robots.txt is a short text file with a long history. It tells automated clients which parts of a site the…
Part 10 looks at what changes when the same technique has to run every day in production. Scrapy makes it very easy to send thousands of requests a minute. Making a spider that a site operator would…
In part 5 we take the ideas from earlier instalments and apply them to a larger, messier site. Scrapy makes it very easy to send thousands of requests a minute. Making a spider that a site operator…