Retry and backoff patterns
Networks fail, servers restart and rate limits kick in. A scraper that gives up at the first error is fragile; one that retries instantly and forever is a nuisance.
Showing 37–40 of 40 articles
Networks fail, servers restart and rate limits kick in. A scraper that gives up at the first error is fragile; one that retries instantly and forever is a nuisance.
Part 3 looks at what changes when the same technique has to run every day in production. Many product pages already contain a clean, structured description of themselves in a script tag meant for…
This is part 3 of our series, and this time we focus on edge cases readers asked about. CSV is the lingua franca of spreadsheets and the most common way scraped data leaves a project. It is also…
Part 3 looks at what changes when the same technique has to run every day in production. Networks fail, servers restart and rate limits kick in. A scraper that gives up at the first error is fragile;…