<?xml version="1.0" encoding="UTF-8"?>
<response><data><post><id>175</id><slug>retry-and-backoff-patterns-part-9-175</slug><title>Retry and backoff patterns, part 9</title><excerpt>This is part 9 of our series, and this time we focus on edge cases readers asked about. Networks fail, servers restart and rate limits kick in. A scraper that gives up at the first error is fragile;…</excerpt><seriesName>Retry and backoff patterns</seriesName><seriesPart>9</seriesPart><sections><item><heading>Introduction</heading><paragraphs><item>This is part 9 of our series, and this time we focus on edge cases readers asked about. Networks fail, servers restart and rate limits kick in. A scraper that gives up at the first error is fragile; one that retries instantly and forever is a nuisance.</item></paragraphs></item><item><heading>A practical approach</heading><paragraphs><item>Time zones matter more than they seem. Store timestamps in UTC and convert only when displaying them.</item><item>Keep configuration such as base URLs, delays and selectors out of your code. Changing a delay should not require a redeploy.</item></paragraphs></item><item><heading>What to watch for</heading><paragraphs><item>Read the site's terms of service before you start. Many sites welcome automated access within limits and say so explicitly; others ask you to use an API instead.</item><item>Before writing a single line of code, look at how the page actually loads its data. Open the network panel, reload, and note which requests return the content you care about. Very often the data you want arrives as JSON from an endpoint you can call directly.</item></paragraphs></item><item><heading>Key takeaway</heading><paragraphs><item>Retry only on errors that can succeed later (timeouts, 429, 5xx), use exponential backoff with jitter, and cap the number of attempts.</item></paragraphs></item></sections><authorId>11</authorId><authorName>Oliver Kim</authorName><authorSlug>oliver-kim</authorSlug><category>Data Engineering</category><tags><item>scrapy</item><item>json</item></tags><publishedAt>2025-11-18T00:00:00.000Z</publishedAt><wordCount>182</wordCount><readingTimeMinutes>2</readingTimeMinutes></post></data></response>
