<?xml version="1.0" encoding="UTF-8"?>
<response><data><post><id>35</id><slug>retry-and-backoff-patterns-part-2-35</slug><title>Retry and backoff patterns, part 2</title><excerpt>In part 2 we take the ideas from earlier instalments and apply them to a larger, messier site. Networks fail, servers restart and rate limits kick in. A scraper that gives up at the first error is…</excerpt><seriesName>Retry and backoff patterns</seriesName><seriesPart>2</seriesPart><sections><item><heading>Introduction</heading><paragraphs><item>In part 2 we take the ideas from earlier instalments and apply them to a larger, messier site. Networks fail, servers restart and rate limits kick in. A scraper that gives up at the first error is fragile; one that retries instantly and forever is a nuisance.</item></paragraphs></item><item><heading>What to watch for</heading><paragraphs><item>Validate the data you extract. Prices without currencies, dates without time zones and empty fields are common sources of silent errors downstream.</item><item>Dynamic pages render content after the initial HTML arrives. A browser automation tool can wait for a specific element to appear, which is far more reliable than sleeping for a fixed amount of time.</item><item>Before writing a single line of code, look at how the page actually loads its data. Open the network panel, reload, and note which requests return the content you care about. Very often the data you want arrives as JSON from an endpoint you can call directly.</item></paragraphs></item><item><heading>Step by step</heading><paragraphs><item>Store raw responses alongside parsed output while you are developing. When a parser breaks, you can replay the saved HTML instead of hitting the site again.</item><item>Always identify your client honestly with a descriptive User-Agent, and keep your request rate modest. A crawler that behaves like a considerate visitor is far less likely to cause problems for the site operator.</item><item>Character encodings still cause trouble. Decode responses using the charset the server declares, and normalise text before comparing or storing it.</item></paragraphs></item><item><heading>Key takeaway</heading><paragraphs><item>Retry only on errors that can succeed later (timeouts, 429, 5xx), use exponential backoff with jitter, and cap the number of attempts.</item></paragraphs></item></sections><authorId>5</authorId><authorName>Chen Patel</authorName><authorSlug>chen-patel</authorSlug><category>Data Engineering</category><tags><item>json</item><item>selenium</item><item>playwright</item><item>ethics</item></tags><publishedAt>2025-04-12T00:00:00.000Z</publishedAt><wordCount>253</wordCount><readingTimeMinutes>2</readingTimeMinutes></post></data></response>
