{"data":{"id":135,"slug":"retry-and-backoff-patterns-part-7-135","title":"Retry and backoff patterns, part 7","excerpt":"Part 7 looks at what changes when the same technique has to run every day in production. Networks fail, servers restart and rate limits kick in. A scraper that gives up at the first error is fragile;…","seriesName":"Retry and backoff patterns","seriesPart":7,"sections":[{"heading":"Introduction","paragraphs":["Part 7 looks at what changes when the same technique has to run every day in production. Networks fail, servers restart and rate limits kick in. A scraper that gives up at the first error is fragile; one that retries instantly and forever is a nuisance."]},{"heading":"A practical approach","paragraphs":["Deduplicate as you go. The same item often appears on several listing pages, and a stable identifier such as a SKU or ID makes duplicates easy to spot.","Keep configuration such as base URLs, delays and selectors out of your code. Changing a delay should not require a redeploy."]},{"heading":"Common pitfalls","paragraphs":["Time zones matter more than they seem. Store timestamps in UTC and convert only when displaying them.","Pagination comes in several shapes: numbered pages, next links, offsets and opaque cursors. Each has trade-offs, and a robust scraper should detect the end of the result set rather than assuming a fixed number of pages."]},{"heading":"Key takeaway","paragraphs":["Retry only on errors that can succeed later (timeouts, 429, 5xx), use exponential backoff with jitter, and cap the number of attempts."]}],"authorId":1,"authorName":"Amelia Okafor","authorSlug":"amelia-okafor","category":"Data Engineering","tags":["json","scrapy"],"publishedAt":"2024-09-04T00:00:00.000Z","wordCount":170,"readingTimeMinutes":2}}