Testing login flows end to end, part 5
For part 5, we revisit the basics with a fresh example and a few lessons learned the hard way. A login form is a small piece of UI with a surprising number of paths through it: success, wrong…
Developer Advocate
Lucas writes about practical data extraction and test automation.
21 articles
For part 5, we revisit the basics with a fresh example and a few lessons learned the hard way. A login form is a small piece of UI with a surprising number of paths through it: success, wrong…
This is part 7 of our series, and this time we focus on edge cases readers asked about. Offset pagination asks for 'items 40 to 60'. Cursor pagination asks for 'the next page after this marker'. The…
This is part 9 of our series, and this time we focus on edge cases readers asked about. When the data you want is several levels deep and the markup has few useful attributes, XPath's ability to…
Part 10 looks at what changes when the same technique has to run every day in production. Lazy loading defers images and sections until they are about to scroll into view. A script that reads the…
In part 10 we take the ideas from earlier instalments and apply them to a larger, messier site. robots.txt is a short text file with a long history. It tells automated clients which parts of a site…
For part 6, we revisit the basics with a fresh example and a few lessons learned the hard way. CSV is the lingua franca of spreadsheets and the most common way scraped data leaves a project. It is…
For part 7, we revisit the basics with a fresh example and a few lessons learned the hard way. Scrapy makes it very easy to send thousands of requests a minute. Making a spider that a site operator…
Running a browser without a window is faster and works on servers, but some problems are much easier to understand when you can watch the browser do its work.
This is part 6 of our series, and this time we focus on edge cases readers asked about. Most flaky Selenium tests are timing problems in disguise. The element exists, just not yet, and a fixed sleep…
Part 6 looks at what changes when the same technique has to run every day in production. Cross-site request forgery tokens are one of the first things that trip up a script trying to log in to a…
A login form is a small piece of UI with a surprising number of paths through it: success, wrong password, locked account, slow responses and expired forms.
Part 5 looks at what changes when the same technique has to run every day in production. Offset pagination asks for 'items 40 to 60'. Cursor pagination asks for 'the next page after this marker'. The…
Offset pagination asks for 'items 40 to 60'. Cursor pagination asks for 'the next page after this marker'. The difference matters as soon as the underlying data changes while you are reading it.
This is part 3 of our series, and this time we focus on edge cases readers asked about. Tables look like the easiest thing on a page to scrape, until you meet merged cells, header rows that repeat…
Part 10 looks at what changes when the same technique has to run every day in production. If a site offers an API for the data you need, scraping its HTML is usually the harder, more fragile and less…
For part 8, we revisit the basics with a fresh example and a few lessons learned the hard way. Many product pages already contain a clean, structured description of themselves in a script tag meant…
For part 9, we revisit the basics with a fresh example and a few lessons learned the hard way. A selector that worked yesterday and fails today is usually a selector that depended on something nobody…
This is part 9 of our series, and this time we focus on edge cases readers asked about. robots.txt is a short text file with a long history. It tells automated clients which parts of a site the…
Networks fail, servers restart and rate limits kick in. A scraper that gives up at the first error is fragile; one that retries instantly and forever is a nuisance.
Part 4 looks at what changes when the same technique has to run every day in production. Infinite scroll replaces the familiar Next button with a page that keeps growing as you move down it, which…
For part 5, we revisit the basics with a fresh example and a few lessons learned the hard way. Running a browser without a window is faster and works on servers, but some problems are much easier to…