Retry and backoff patterns, part 8
For part 8, we revisit the basics with a fresh example and a few lessons learned the hard way. Networks fail, servers restart and rate limits kick in. A scraper that gives up at the first error is…
Showing 25–36 of 59 articles
For part 8, we revisit the basics with a fresh example and a few lessons learned the hard way. Networks fail, servers restart and rate limits kick in. A scraper that gives up at the first error is…
In part 2 we take the ideas from earlier instalments and apply them to a larger, messier site. Networks fail, servers restart and rate limits kick in. A scraper that gives up at the first error is…
Part 6 looks at what changes when the same technique has to run every day in production. Cross-site request forgery tokens are one of the first things that trip up a script trying to log in to a…
Part 6 looks at what changes when the same technique has to run every day in production. Logging in once and staying logged in is a matter of keeping the cookies the server gives you and sending them…
This is part 8 of our series, and this time we focus on edge cases readers asked about. If a site offers an API for the data you need, scraping its HTML is usually the harder, more fragile and less…
In part 9 we take the ideas from earlier instalments and apply them to a larger, messier site. Lazy loading defers images and sections until they are about to scroll into view. A script that reads…
In part 9 we take the ideas from earlier instalments and apply them to a larger, messier site. Logging in once and staying logged in is a matter of keeping the cookies the server gives you and…
Cross-site request forgery tokens are one of the first things that trip up a script trying to log in to a site: the form posts fine in a browser but fails with a 403 from code.
A login form is a small piece of UI with a surprising number of paths through it: success, wrong password, locked account, slow responses and expired forms.
For part 9, we revisit the basics with a fresh example and a few lessons learned the hard way. Most flaky Selenium tests are timing problems in disguise. The element exists, just not yet, and a fixed…
Offset pagination asks for 'items 40 to 60'. Cursor pagination asks for 'the next page after this marker'. The difference matters as soon as the underlying data changes while you are reading it.
In part 6 we take the ideas from earlier instalments and apply them to a larger, messier site. A one-off script grows into a pipeline the moment someone asks for the data again next week. A little…