Testing login flows end to end, part 6
This is part 6 of our series, and this time we focus on edge cases readers asked about. A login form is a small piece of UI with a surprising number of paths through it: success, wrong password,…
Staff Writer
Chen writes about practical data extraction and test automation.
27 articles
This is part 6 of our series, and this time we focus on edge cases readers asked about. A login form is a small piece of UI with a surprising number of paths through it: success, wrong password,…
In part 2 we take the ideas from earlier instalments and apply them to a larger, messier site. When the data you want is several levels deep and the markup has few useful attributes, XPath's ability…
Part 9 looks at what changes when the same technique has to run every day in production. Tables look like the easiest thing on a page to scrape, until you meet merged cells, header rows that repeat…
This is part 2 of our series, and this time we focus on edge cases readers asked about. A selector that worked yesterday and fails today is usually a selector that depended on something nobody…
For part 3, we revisit the basics with a fresh example and a few lessons learned the hard way. Scrapy makes it very easy to send thousands of requests a minute. Making a spider that a site operator…
Part 2 looks at what changes when the same technique has to run every day in production. Logging in once and staying logged in is a matter of keeping the cookies the server gives you and sending them…
In part 5 we take the ideas from earlier instalments and apply them to a larger, messier site. Cross-site request forgery tokens are one of the first things that trip up a script trying to log in to…
Part 10 looks at what changes when the same technique has to run every day in production. Logging in once and staying logged in is a matter of keeping the cookies the server gives you and sending…
In part 7 we take the ideas from earlier instalments and apply them to a larger, messier site. Infinite scroll replaces the familiar Next button with a page that keeps growing as you move down it,…
Part 3 looks at what changes when the same technique has to run every day in production. robots.txt is a short text file with a long history. It tells automated clients which parts of a site the…
In part 10 we take the ideas from earlier instalments and apply them to a larger, messier site. Many product pages already contain a clean, structured description of themselves in a script tag meant…
Part 9 looks at what changes when the same technique has to run every day in production. A well-behaved API tells you how many requests you have left before it starts saying no. Reading those headers…
Part 5 looks at what changes when the same technique has to run every day in production. Before you crawl a site link by link, check whether it already publishes a list of every page it wants found.…
In part 8 we take the ideas from earlier instalments and apply them to a larger, messier site. Offset pagination asks for 'items 40 to 60'. Cursor pagination asks for 'the next page after this…
For part 5, we revisit the basics with a fresh example and a few lessons learned the hard way. A selector that worked yesterday and fails today is usually a selector that depended on something nobody…
This is part 5 of our series, and this time we focus on edge cases readers asked about. When the data you want is several levels deep and the markup has few useful attributes, XPath's ability to…
This is part 9 of our series, and this time we focus on edge cases readers asked about. A one-off script grows into a pipeline the moment someone asks for the data again next week. A little structure…
Many product pages already contain a clean, structured description of themselves in a script tag meant for search engines. Reading it is often more reliable than parsing the visible HTML.
In part 2 we take the ideas from earlier instalments and apply them to a larger, messier site. Networks fail, servers restart and rate limits kick in. A scraper that gives up at the first error is…
For part 9, we revisit the basics with a fresh example and a few lessons learned the hard way. Most flaky Selenium tests are timing problems in disguise. The element exists, just not yet, and a fixed…
For part 2, we revisit the basics with a fresh example and a few lessons learned the hard way. CSV is the lingua franca of spreadsheets and the most common way scraped data leaves a project. It is…
For part 8, we revisit the basics with a fresh example and a few lessons learned the hard way. When the data you want is several levels deep and the markup has few useful attributes, XPath's ability…
This is part 10 of our series, and this time we focus on edge cases readers asked about. Infinite scroll replaces the familiar Next button with a page that keeps growing as you move down it, which…
For part 4, we revisit the basics with a fresh example and a few lessons learned the hard way. Many product pages already contain a clean, structured description of themselves in a script tag meant…
Part 2 looks at what changes when the same technique has to run every day in production. Scrapy makes it very easy to send thousands of requests a minute. Making a spider that a site operator would…
This is part 3 of our series, and this time we focus on edge cases readers asked about. CSV is the lingua franca of spreadsheets and the most common way scraped data leaves a project. It is also…
Part 3 looks at what changes when the same technique has to run every day in production. Networks fail, servers restart and rate limits kick in. A scraper that gives up at the first error is fragile;…